Virtual reality and 3D printing improve preoperative visualization of 3D liver reconstructions—results from a preclinical comparison of presentation modalities and user’s preference

Introduction

In complex liver surgery personalized resection planning is of high importance due to the variability of individual hepatic anatomy (1-3). Hence it is of utmost importance to know the location of the tumor within the liver as well as its positional relationship to the blood vessels. Furthermore, it is necessary to anticipate the sustained vascular and biliary supply and drainage as well as the size of the future liver remnant (1,3,4). To retrieve all this information out of a two dimensional slice-based computer tomography (CT) or magnetic resonance imaging (MRI) is difficult and a mentally demanding task, which needs years of experience and training (5). Three dimensional (3D) reconstructions of the liver parenchyma, the vessels and the tumor have been available for surgical planning for more than two decades (3). Already in 2000, Lamadé et al. were able to show that color-coded 3D reconstructions can increase the precision in tumor localization by 37% compared to the standard 2D CT scan and Lang et al. showed a change of the initial 2D based resection plan due to computer-assisted risk analysis in 33% of their cases (3,6). Within the next 20 years the importance of these 3D reconstructions increased progressively; thus, Banz et al. state that 3D reconstructions are ‘imperative in major, complex liver resections’ and in Japan the universal healthcare insurance covers the additional costs for a preoperative reconstruction since 2012 (7,8). Also several other study groups were able to show that 3D reconstruction is preferable to the standard 2D scans in centrally located hepatocellular carcinoma and bilateral colorectal metastases (9,10).

In the literature, there is no information regarding the ‘optimal’ presentation modality of 3D liver models. Since the beginning of 3D reconstruction and its presentation on 2D monitors as interactive 3D PDFs, technological progress enables not only live-size 3d printed models (PR) but also virtual presentations forms with augmented or virtual reality (AR/VR) devices (11-13). These new technologies (PR and VR/AR) have not been well investigated, especially in comparison with the established presentation mode, even though they have several benefits especially in depth perception compared to a 3D model on a 2D monitor (14). This is the first study focusing on the presentation format of the 3D datasets in hepatic surgery including latest technologies with regard to anatomical orientation and personal preferences in a high volume center for liver surgery. We present the following article in accordance with the MDAR checklist (available at http://dx.doi.org/10.21037/atm-21-512).

Methods

Patient selection and participants

The data of 20 patients with various liver pathologies, who underwent liver surgery in our department, where retrospectively selected for this study (Table 1). The selection of patients’ ensured a variety of resection types (minor-/major resection/extended resection/vessel reconstruction). In addition, another inclusion criterion was the availability of a preoperative, high quality CT scan.

Table 1 Characteristics of selected patients (n=20)
Full table

A total of 30 participants of different training levels [medical students (n=10), residents (n=10), fellows (n=5) and HPB experts (n=5)] were included in this study.

Three-dimensional models

For three-dimensional reconstruction, a semi-automatic, server-based program (Synapse 3D, FUJIFILM Europe GmbH, Düsseldorf) was used. The three-dimensional visualizations were based on the preoperative computed tomography (CT) scans. A surgical resident of the department (FH), who has five years of clinical experience in a surgical high volume liver center and three years of experience using the reconstruction software, performed all reconstructions (15). These reconstructions can be directly exported from the software to a 3D PDF as well as to stereolithography (STL) files, which are required for the 3D PR models as well as the VR application. Therefore, all examination modalities are based on the same reconstruction (Figure 1).

Figure 1 3D reconstruction of a patient’s individual liver with three colorectal metastases located in segment 5 and 8 as a 3D PR model (A), as 3D PDF (B) and in VR (C).

Examination modalities

• PDF: the 3D PDF is directly created by the software via an export function. Within the PDF the liver model can be rotated around a vertical or horizontal axis via the keyboard or manual in all directions and differently scaled using the mouse. In addition, the different structures (e.g., tumor) can be displayed solid, transparently or hidden.
• PR: The 3D models were printed by Cella Medical Solutions (Murcia, Spain) using the STL files, which are generated by the reconstruction software as previously described (16). For the parenchyma a transparent material (polyurethane rubber) is used, so that the colored structures (acrylonitrile butadiene styrene with pigments) in the inside (e.g., vessels, tumor) can be seen. The solid and inflexible liver can be held and turned by hand enabling haptic feedback.
• VR: The specifically developed application, using the game engine Unity (version 2019.2.14f1, Unity Technologies, San Francisco, California, USA) enables an easy upload of STL files via drag and drop and then automatically generates the VR 3D model. The application provides native support for all OpenVR compatible VR headsets. The 3D liver model can be moved, rotated around a vertical or horizontal axis as well as manual by means of the VR controller. Additionally, the model size can be scaled and the structures can be continuously adjusted to different degrees of transparency (solid to hidden) (17).

Equipment, tests and questionnaires

Table 2 gives an overview of all used tools.

Table 2 Compilation of used equipment, test and questionnaires
Full table

Study protocol

The study consisted of two sessions (Figure 2). At the beginning of the first session a standardized study description and explanation was given to all participants, followed by two tests regarding their spatial thinking and seeing [Hose figure test and Titmus test (18)] as well as the pre-simulator sickness questionnaire (21). After a short introduction of the presentation modalities the participants had to name all tumor-bearing segments for each liver model. The time needed for the assignment was recorded. The participants received no feedback regarding the correctness of their answers. To minimize a selection or recognition bias the modality was changed after every fifth liver model (= block) and each participant had an individually randomized order of presentation modality and liver ID. The first session was terminated after the first 30 models were processed. The post-simulator sickness questionnaire was completed after the first VR block.

Figure 2 Study design. SSQ, Simulator sickness questionnaire; SUS, System Usability Scale; UEQ, User Experience Questionnaire.

The second session started with the assignment of the remaining 30 models in the same way as in session 1. At the end the participants were asked to answer several questionnaires regarding usability and personal preferences (19,20).

Tumor assignment

To establish a standardized correct answer, an internationally well-known expert in the field of liver surgery (HL) with many years of experience regarding hepatic tumor assignment evaluated the CT scans of all patients prior to the study. The assigned tumor-bearing segments based on the 2D data were then reevaluated by the expert using the different 3D modalities. The assignment was based on the Couinaud classification (22).

Statistical analysis

All statistical analyses were conducted using IBM SPSS Statistics 26 (IBM, Armonk, NY, USA) and R v. 3.6.2 [R Core Team (2019), R: A language and environment for statistical computing, R Foundation for Statistical Computing, Vienna, Austria]. The Mann-Whitney U test was used for continuous, unrelated data and the Friedman test for continuous related data. Spearman’s rank correlation coefficient was used for correlation analysis. The impact of different variables on time and validity was analyzed using mixed models with liver ID and participant ID as random variables utilizing the lme4 R package (23). Time was analyzed on log scale using the lmer function. Validity was measured as proportion of correctly detected liver segments and analyzed using the glmer function with binomial family and the number of segments as weight. A correctness ratio was calculated by dividing the number of correct assigned segments through the number of segments. The local significance level was set to 5%. Correction for multiple testing was not performed due to the exploratory character of the study.

Ethics

The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the regional Ethics Committee of the Medical Association of Rhineland-Palatinate, Germany (NO.: 2019-14449) and informed consent was taken from all individual participants. The study participation was voluntary and an expense allowance was offered to all participants.

Results

Demographics

Gender distribution over all qualification levels was even (m/f =15/15): medical students (m/f =3/7), residents (m/f =5/5), fellows (m/f =4/1) and HPB experts (n=5; m/f =3/2). Righthandedness was predominant (n=27).

Tumor assignment

Random effects for the correctness ratio reveal a low variance between participants (var =0.041) and high variance between livers (var =0.996). This implies that tumor assignment of some liver models was more difficult than others for all participants regardless of the level of education.

Participants named significantly more correct segments, when the liver model was presented in VR (P=0.040) or PR (P=0.036) compared to PDF. No significant difference was seen between VR and PR (P=0.964) (Figure 3). Also the level of qualification had an influence on the correctness of tumor assignment. HPB experts were significantly better than students (P<0.001) by achieving a mean ratio of 0.87 (SD 0.12) compared to students 0.83 (SD 0.15) analyzed over all modalities. Table 3 gives a detailed summary of the different correctness ratios categorized by modality and level of education.

Figure 3 Mean number of correctly named segments. The diamond indicates the mean number of correct segments for all participants and all liver models grouped by presentation modality with a statistically significant difference between VR and PR compared to PDF. The whiskers represent two standard deviations above and below the mean.

Table 3 Comparison of mean achieved correctness ratios for assessment of tumor-bearing segments for different education levels
Full table

In contrast to the time analysis, the result of the hose figure test had a significant positive influence on the correctness ratio (P=0.011; Figure 4A), indicating that participants with higher abilities for 3D thinking are better in the assessment of tumor-bearing segments regardless of presentation modality. Also better Titmus test results had a positive influence on the correctness ratio, but fail to reach statistically significance (P=0.290; Figure 4B) for univariate analysis. However, in a multivariate analysis, test results for stereoacuity test show next to presentation modality, qualification of participants and the hose figure test a significant influence (P=0.019). Gender had no influence on assessment quality (P=0.393). Compared to time analysis, where participants were significantly faster in the second study session, this had no influence on the assessment of tumor-bearing segments (P=0.487), implying that short “training” during the first session improves average speed but not quality.

Figure 4 Influence of spatial thinking and seeing on the correctness ratio. (A) The diagram displays the influence of the test results of the mental rotation test on the correctness ratio for each modality. (B) The diagram displays the influence of the stereoacuity test results on the correctness ratio for each modality.

In addition, analysis of time to correctness ratio reveals a low negative correlation (P<0.001, rho =−0.229). This indicates that it is in general not helpful to prolong the assessment time to achieve a higher assessment quality, meaning that the participants either identify the tumor-bearing segments quickly and correct or have difficulties to name the affected segments regardless of assessment time.

Time analysis

Random effects for time show a moderate variance between participants (var =0.12) and between livers (var =0.27).

Comparing the time needed to assign all tumor-bearing segments 3D PR was significantly faster than PDF (P<0.001) or VR (P<0.001). No difference was seen between PDF and VR (P=0.793). In conclusion, the total time needed to assign the tumor location for liver models was significantly shorter for the 3D PR (509±156 s) compared to the VR (702±248 s) and the PDF (734±281 s) (P<0.01).

Regardless of the presentation modality, HPB experts were significantly faster than students (P=0.014; Figure 5). No statistical significant difference was achieved between the other subgroups (P>0.05).

Figure 5 Boxplot. The diagram displays time needed for tumor assignment for each modality grouped by qualification level. The assessment time is displayed as log scale. (*P<0.05).

Univariate analysis shows no relevant influence on time neither for the test result of spatial thinking nor of visual spatial ability or gender (P>0.05). Multivariate analysis confirmed a statistical relevant influence of study session and modality on time (P<0.001).

Questionnaires and tests

The new VR application was rated best regarding usability for preoperative liver surgery planning. Ninety percent (n=27) stated that this application can positively influence the operation planning. The 3D PR was rated as helpful by 73.3% (n=22) and the 3D PDF by 46.7% (n=14). In addition, 73.3% (n=22) preferred VR over the other modalities and 26.7% (n=8) favored the 3D PR. The selection of preferred modality was influenced by the level of education. So favored 100% (n=10) of students and 80% (n=8) of residents in comparison to 40% (n=2) of fellows and HPB experts the VR application. The PR on the other hand was the preferred modality in the group of fellows and HPB experts with 60% (n=3) compared to 0% of students and 20% (n=2) of residents, respectively. This results in a statistically significant difference between experienced (fellows and HPB experts) and rather inexperienced (students and residents) (P=0.028).

Twenty-five participants (83.4%) rated the VR application and 22 (73.4%) the 3D PR model as good or even excellent in the System Usability Scale, compared to just 10 (33.3%) for the 3D PDF. The mean score values [SD] were: VR 83.25 [13.99], 3D PR 80.25 [13.22] and 3D PDF 61.92 [21.57].

The mean score and subscore of the User Experience Questionnaire are displayed in Figure 6. Compared to the benchmark values of Schrepp et al. the VR application achieves excellent results (in the range of the 10% best results) and the 3D PR models are in all categories above average (25% of results better, 50% of results worse) whereas the 3D PDF is rated below average (in the range of the 25% worst results) (24).

Figure 6 Boxplots of subscores and score values of the User Experience Questionnaire (UEQ) for the three different presentation modalities. Values above 0.8 are defined as positive evaluation results (red dotted line).

Comfort of the VR glasses was rated good or very good by 93.4% (n=28) of participants. Results of the Simulator Sickness Questionnaire show that the use of the VR application induced no relevant vegetative sight effects [mean (SD): total score_prae 58.49 (67.92) vs. total score_post 89.50 (148.19); P=0.475].

Discussion

In complex liver surgery a profound knowledge of the patient’s individual hepatic anatomy is of particular importance. 3D reconstructions of the liver have been shown to be helpful for resection planning. Several studies have demonstrated that 3D visualizations have advantages over 2D scans (5,25,26). However, the role of the ‘presentation modality’ for these 3D visualizations is not well investigated. Our results indicate that the precision of tumor localization is higher when the reconstruction is presented as a PR model or in VR, while the shortest assessment times were recorded for the 3D PR. In addition, the majority of participants see the greatest potential for a positive influence on resection planning in the recently developed immersive VR application and prefer it over the other modalities.

3D visualizations have a high potential to enhance patient safety and surgical precision in the field of liver surgery. The meta-analysis of Zhang et al. could demonstrate that the use of 3D visualizations results in reduced intraoperative blood loss, postoperative complications, operation time, and hospitalization time, as well as a faster recovery of postoperative liver function, and lower tumor recurrence on short-term follow-up (27). In addition, Bangeas et al. conclude in their meta-analysis regarding the role of 3D printed models that 3D PR has an additional benefit over ‘just’ a 3D visualization (e.g., 3D PDF) for preoperative planning and as an educational tool. In contrast, Kong et al. found no difference between 3D PDF and 3D PR regarding anatomical education (28). However, most studies published in literature to date compare 2D CT scans with 3D PDFs either with or without an additional 3D PR (29-31). Yang et al. were the first group to compare different 3D modalities (3D PDF and 3D PR) to conventional 2D CT scans regarding tumor assignment and surgical resection planning, favoring the 3D PR reconstructions (32). The 3D VR model used in this study has first and only been published by our study group as a result of an interprofessional collaboration between surgeons and computer scientists (17). The often-used term of ‘virtual model/virtual resection’ used in other publications referrers to the standard 3D visualization on a 2D or 3D monitor, not to an immersive VR environment with VR headsets (26,33,34). This study is a first step towards an objective evaluation of currently available 3D presentation modalities factoring in newest technologies to improve anatomical education and the safety of liver surgery.

The localization of the tumor(s) within the liver is the first step towards an adequate resection planning. Therefore, the primary endpoint of the study was to determine, whether participants achieve a higher score in one of the three modalities. Statistical analyzes revealed a superiority of 3D PR models and 3D VR models over reconstructions presented as 3D PDFs. This result applies to all levels of education with the highest impact on students and the lowest impact on HPB experts. The main reason for this finding is the loss of depth perception through the projection of a 3D object on a 2D screen in the case of 3D PDFs (35). However experienced surgeons use secondary visual references to interpret non-stereo depth cues to compensate for the loss of the third dimension (36,37). The PR models can be explored using the haptic and visual sense (32). It is well investigated that especially the involvement of the haptic sense plays an important role due to the enrichment of visual perception (38). The VR model, although lacking the haptic sense input of the PR model, allows depth perception and binocular field of view, creating an immersive 3D experience (14,39). Furthermore, the ability to alter the model in size and transparency seems to compensate for the loss of haptic sense. This is supported by the correctness of tumor assignment, the results of the user experience and usability questionnaires as well as the stated subjective preferences of the participants. Thus, based on objective and subjective parameters, 3D PR models and the 3D VR models are the preferred modalities within our study population.

The time needed for the assessment of tumor-bearing segments was used as a surrogate parameter for the ability to adequately use the modality as well as the primary orientation within the 3D model. The results revealed that the 3D PR models are mostly superior to the other modalities regardless of educational level. This can be explained by the fact that the PR models are the most ‘natural’ modality, which can be explored without any technical aids. More surprisingly is the fact that with the VR application, which was unknown to the participants, assessment time was even a little shorter than with the well established 3D PDF presentation modality (40). This indicates that the new technology is easy to use and does not prolong the assessment. This thesis is also supported by the excellent evaluation results for usability (21). The quality of anatomical orientation can’t be estimated by solely analyzing the time. Nevertheless, a negative correlation between time and correctness ratio could imply a better anatomical orientation through shorter assessment time. This negative correlation can be seen for all educational levels and all modalities, not explicitly favoring one presentation format. We could show that assessment time for tumor localization is faster with 3D PR while the developed VR technology is comparable to the established 3D PDF format.

The question, which 3D modality is the best for preoperative planning, also needs to factor in production time, acquisition costs as well as availability. The segmentation of the 2D scan stands at the beginning of all 3D presentation formats. Regardless of the provider (external or local) the reconstruction is always initially presented as 3D PDF, therefore the 3D PDF is the fastest and most cost-efficient modality (15). The printing process of 3D models can take days and is associated with high costs (25,41). Also the VR application - or rather the equipment for it is costly. But in contrast to the PR models, which are invoiced per case, the procurement costs for the VR devices are only incurred once (17). Furthermore, no relevant additional time is needed to create the 3D VR model due to a drag and drop application for STL files. These factors need to be considered before using a modality for preoperative surgical planning. When it comes to education, which is the second most common application for 3D models, also sustainability and availability is of utterly importance (41). Here the breakable PR model is in clear disadvantage to the 3D PDF and the VR technology. A particular advantage of the PDF is that it can be explored on any regular computer. In addition, the PDF and the VR are not locally bound and can therefore be assessed simultaneously by different trainees in different locations. Especially the VR application enables collaborative work over distance with the potential of a multiuser mode (42). Therefore, the choice of the modality is largely dependent on the number of cases, for which a reconstruction is required, the time pressure and the application area (surgical planning/education).

A limitation of the study is a potential recognition effect when the liver model is presented the second or third time to the participant. To minimize an effect on the statistical analysis, the order of liver models as well as the order of presentation modalities was randomly selected to ensure a unique order for each participant. Furthermore, the segmentation of the liver into its segments is to some point subjective, especially if the anatomy is altered due to a malignant process. Nevertheless, surgical documentation and medical communication require the assignment of tumor-bearing segments. Therefore, we considered an expert in the field of liver segmentation and surgery (HL), to be the best way to establish a standardized correct answer.

Conclusions

This is the first preclinical study about anatomical orientation and personal preferences of three different 3D presentation modalities in liver surgery. We have shown that each modality has its advantages and disadvantages. While the PDF is most cost effective, the precision of tumor localization is higher when the reconstruction is presented as a PR model or in VR, while the shortest assessment times where recorded for the 3D PR. Nevertheless, the majority of participants preferred the VR application over the other modalities and believed in a positive influence on resection planning with this modality. Further clinical evaluation is necessary to evaluate the three presentation modalities in the context of actual preoperative, surgical planning.

Acknowledgments

Funding: The study was supported by the German Federal Ministry of Education and Research (FKZ: 16SV8057 “AVATAR”). Intramural funding from the Medical Centre of the Johannes Gutenberg University of Mainz provided further financial support for the project.

Footnote

Reporting Checklist: The authors have completed the MDAR checklist. Available at http://dx.doi.org/10.21037/atm-21-512

Data Sharing Statement: Available at http://dx.doi.org/10.21037/atm-21-512

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/atm-21-512). Dr. FH reports grants from German Federal Ministry of Education and Research, other from Medical Centre of the Johannes Gutenberg University of Mainz, during the conduct of the study. Dr. PS, Dr. CH, Dr. BP, Dr. WK, Dr. HL, and Dr. TH report grants from German Federal Ministry of Education and Research, during the conduct of the study. AP has no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the regional Ethics Committee of the Medical Association of Rhineland-Palatinate, Germany (NO.: 2019-14449) and informed consent was taken from all individual participants.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Selle D, Preim B, Schenk A, et al. Analysis of vasculature for liver surgical planning. IEEE Trans Med Imaging 2002;21:1344-57. [Crossref] [PubMed]
2. Catalano OA, Singh AH, Uppot RN, et al. Vascular and biliary variants in the liver: implications for liver surgery. Radiographics 2008;28:359-78. [Crossref] [PubMed]
3. Lang H, Radtke A, Hindennach M, et al. Impact of virtual tumor resection and computer-assisted risk analysis on operation planning and intraoperative strategy in major hepatic resection. Arch Surg 2005;140:629-38; discussion 38. [Crossref] [PubMed]
4. Lang H, Radtke A, Liu C, et al. Improved assessment of functional resectability in repeated hepatectomy by computer-assisted operation planning. Hepatogastroenterology 2005;52:1645-8. [PubMed]
5. Yeo CT, MacDonald A, Ungi T, et al. Utility of 3D Reconstruction of 2D Liver Computed Tomography/Magnetic Resonance Images as a Surgical Planning Tool for Residents in Liver Resection Surgery. J Surg Educ 2018;75:792-7. [Crossref] [PubMed]
6. Lamadé W, Glombitza G, Fischer L, et al. The impact of 3-dimensional reconstructions on operation planning in liver surgery. Arch Surg 2000;135:1256-61. [Crossref] [PubMed]
7. Banz VM, Muller PC, Tinguely P, et al. Intraoperative image-guided navigation system: development and applicability in 65 patients undergoing liver surgery. Langenbecks Arch Surg 2016;401:495-502. [Crossref] [PubMed]
8. Oshiro Y, Ohkohchi N. Three-Dimensional Liver Surgery Simulation: Computer-Assisted Surgical Planning with Three-Dimensional Simulation Software and Three-Dimensional Printing. Tissue Eng Part A 2017;23:474-80. [Crossref] [PubMed]
9. Fang CH, Tao HS, Yang J, et al. Impact of three-dimensional reconstruction technique in the operation planning of centrally located hepatocellular carcinoma. J Am Coll Surg 2015;220:28-37. [Crossref] [PubMed]
10. Aloia TA. Extending the Limits of Resection for Colorectal Liver Metastases: Bilateral Disease. J Gastrointest Surg 2017;21:193-5. [Crossref] [PubMed]
11. Maresceaux J, Soler L, Ceulemans R, et al. Image fusion, virtual reality, robotics and navigation. Effects on surgical practice. Chirurg 2002;73:422-7. [Crossref] [PubMed]
12. Sauer IM, Queisner M, Tang P, et al. Mixed Reality in Visceral Surgery: Development of a Suitable Workflow and Evaluation of Intraoperative Use-cases. Ann Surg 2017;266:706-12. [Crossref] [PubMed]
13. Saito Y, Sugimoto M, Imura S, et al. Intraoperative 3D Hologram Support With Mixed Reality Techniques in Liver Surgery. Ann Surg 2020;271:e4-e7. [Crossref] [PubMed]
14. Heinrich F, Rohde S, Huber T, et al. VR-basierte Interaktion mit 3D-Organmodellen zur Planung und Simulation laparoskopischer Eingriffe. Proceedings of the Annual Meeting of the German Society of Computer- and Robot-Assisted Surgery (CURAC); Leipzig, Germany 2018.
15. Paschold M, Huettl F, Kneist W, et al. Local, semi-automatic, three-dimensional liver reconstruction or external provider? An analysis of performance and time expense. Langenbecks Arch Surg 2020;405:173-9. [Crossref] [PubMed]
16. Huber T, Huettl F, Tripke V, et al. Experiences With Three-dimensional Printing in Complex Liver Surgery. Ann Surg 2021;273:e26-7. [Crossref] [PubMed]
17. Boedecker C, Huettl F, Saalfeld P, et al. Using virtual 3D-models in surgical planning: workflow of an immersive virtual reality application in liver surgery. Langenbecks Arch Surg 2021;406:911-5. [Crossref] [PubMed]
18. Test für medizinische Studiengänge und Eignungstest für das Medizinstudium III. Goettingen: Hogrefe Verlag GmbH & Co. KG; 2019.
19. Bangor A, Kortum PT, Miller JT. An Empirical Evaluation of the System Usability Scale. Int J Hum Comput Interact 2008;24:574-94. [Crossref]
20. Kennedy RS, Lane NE, Berbaum KS, et al. Simulator Sickness Questionnaire: An Enhanced Method for Quantifying Simulator Sickness. Int J Aviat Psychol 1993;3:203-20. [Crossref]
21. Laugwitz B, Held T, Schrepp M. Construction and Evaluation of a User Experience Questionnaire. Available online: https://link.springer.com/chapter/10.1007/978-3-540-89350-9_6
22. Couinaud C. Le foie: études anatomiques et chirurgicales. Masson; 1957.
23. Bates D, Mächler M, Bolker B, et al. Fitting Linear Mixed-Effects Models Using lme4. J Stat Softw 2015;67:48. [Crossref]
24. Schrepp M, Hinderks A, Thomaschewski J. Construction of a Benchmark for the User Experience Questionnaire (UEQ). International Journal of Interactive Multimedia and Artificial Intelligence 2017;4:40-4. [Crossref]
25. Yao R, Xu G, Mao SS, et al. Three-dimensional printing: review of application in medicine and hepatic surgery. Cancer Biol Med 2016;13:443-51. [Crossref] [PubMed]
26. Fang C, An J, Bruno A, et al. Consensus recommendations of three-dimensional visualization for diagnosis and management of liver diseases. Hepatol Int 2020;14:437-53. [Crossref] [PubMed]
27. Zhang W, Fang C. Meta-Analysis of efficacy comparison between diagnosis and treatment of primary hepatocellular carcinoma treated by 3D visualization technology and 2D imaging technology. Chin J Pract Surg 2019;39:44-50.
28. Bangeas P, Tsioukas V, Papadopoulos VN, et al. Role of innovative 3D printing models in the management of hepatobiliary malignancies. World J Hepatol 2019;11:574-85. [Crossref] [PubMed]
29. Zeng N, Yang J, Xiang N, et al. Application of 3D visualization and 3D printing in individualized precision surgery for Bismuth-Corlette type III and IV hilar cholangiocarcinoma. Nan Fang Yi Ke Da Xue Xue Bao 2020;40:1172-7. [PubMed]
30. Kong X, Nie L, Zhang H, et al. Do 3D Printing Models Improve Anatomical Teaching About Hepatic Segments to Medical Students? A Randomized Controlled Study. World J Surg 2016;40:1969-76. [Crossref] [PubMed]
31. Irtan S, Hervieux E, Boutroux H, et al. Preoperative 3D reconstruction images for paediatric tumours: Advantages and drawbacks. Pediatr Blood Cancer 2021;68:e28670 [Crossref] [PubMed]
32. Yang T, Lin S, Xie Q, et al. Impact of 3D printing technology on the comprehension of surgical liver anatomy. Surg Endosc 2019;33:411-7. [Crossref] [PubMed]
33. Mutter D, Dallemagne B, Bailey C, et al. 3D virtual reality and selective vascular control for laparoscopic left hepatic lobectomy. Surg Endosc 2009;23:432-5. [Crossref] [PubMed]
34. Li A, Tang R, Rong Z, et al. The Use of Three-Dimensional Printing Model in the Training of Choledochoscopy Techniques. World J Surg 2018;42:4033-8. [Crossref] [PubMed]
35. Sørensen SM, Savran MM, Konge L, et al. Three-dimensional versus two-dimensional vision in laparoscopy: a systematic review. Surg Endosc 2016;30:11-23. [Crossref] [PubMed]
36. Patrzyk M, Klee M, Stefaniak T, et al. Randomized study of the influence of two-dimensional versus three-dimensional imaging using a novel 3D head-mounted display (HMS-3000MT) on performance of laparoscopic inguinal hernia repair. Surg Endosc 2018;32:4624-31. [Crossref] [PubMed]
37. Fergo C, Burcharth J, Pommergaard HC, et al. Three-dimensional laparoscopy vs 2-dimensional laparoscopy with high-definition technology for abdominal surgery: a systematic review. Am J Surg 2017;213:159-70. [Crossref] [PubMed]
38. Wijntjes MW, Volcic R, Pont SC, et al. Haptic perception disambiguates visual perception of 3D shape. Exp Brain Res 2009;193:639-44. [Crossref] [PubMed]
39. Ong CS, Deib G, Yesantharao P, et al. Virtual Reality in Neurointervention. J Vasc Interv Neurol 2018;10:17-22. [PubMed]
40. Merril JR, Preminger GM, Babayan R, et al. Surgical simulation using virtual reality technology: design, implementation, and implications. Surg Technol Int 1994;3:53-60. [PubMed]
41. Pietrabissa A, Marconi S, Negrello E, et al. An overview on 3D printing for abdominal surgery. Surg Endosc 2020;34:1-13. [Crossref] [PubMed]
42. Chheang V, Saalfeld P, Huber T, et al. Collaborative Virtual Reality for Laparoscopic Liver Surgery Training. Available online: https://ieeexplore.ieee.org/document/8942324