Development and clinical applications of glasses-free three-dimensional (3D) display technology for thoracoscopic surgery
Minimally invasive surgery represents the future of modern surgery. In particular, endoscope technology is the most important minimally invasive technique in the field of surgery. In 1990, Lewis et al. firstly described the use of video imaging system in the diagnosis and treatment of chest diseases, marking the beginning of a new era of video-assisted thoracoscopic surgery (VATS). After more than 20 years of development, VATS has been widely applied in thoracic surgery as a minimally invasive surgical technique. In 2012, the US National Comprehensive Cancer Network (NCCN) included the VATS diagnosis and treatment technology in its lung cancer diagnosis and treatment guidelines. However, VATS adopts traditional two-dimensional (2D) technology and has certain limitations: (I) it lacks a three-dimensional (3D) sense, and the operator can hardly determine the distance between the position where the operation device arrives and the target region; and (II) the operator needs to receive long-term training before he/she can easily shift between 2D and 3D images. With the development of imaging technology, 3D high-definition video technology has also been used in thoracoscopic surgery. The operator and visitors can wear auxiliary 3D glasses and watch stereoscopic surgical images on 3D TVs, as done in a robotic surgery. It has the following advantages over the 2D systems: (I) the images are clearer after zooming in: the maximum magnification is 4× for a 2D system and 10× for a 3D system, and the latter can display the anatomical structures more clearly; (II) the anatomy of living tissues and structures can be more vividly visualized, with marked 3D sense and depth; it can clearly display the levels of the tubular structures of blood vessels and bronchi, the stereoscopic positioning of the mediastinal lymph nodes, and their relationships with the surrounding tissues; (III) thanks to its clear stereoscopic sense, the surgical operations including dissection, dissociation, and suturing can be displayed more precisely, with clear indications of depth and direction, which minimizes the occurrence of surgical trauma; (IV) it has shorter learning curve for beginners; for experienced thoracic surgeons, it also shortens the reaction time in structure identification (1,2). However, the currently available auxiliary 3D technology still has some drawbacks: (I) the brightness of the light decreases after the light passes through the polarizer, resulting in relatively dark visual field; (II) for surgeons who are not accustomed to wearing glasses, the glasses often cause various uncomfortable feelings: for example, water vapor causes the lenses to become blurred, and wearing the glasses for long time causes discomfort to the nose and ears; and (III) all the 3D videos are still 2D signals during playback and other processes, which is not conducive to learning and experience dissemination. Therefore, many predecessors are also developing glasses-free 3D display systems, with an attempt to avoid the use of auxiliary 3D glasses. In 2012, multi-view glasses-free 3D display system has been tried in clinical settings (3). However, so far, few studies have analyzed the clinical application of glasses-free 3D endoscopic techniques, mainly due to the poor image quality and narrow display angle of some early display systems. Also, the mainstream LCD panel manufacturers mainly adopt parallax barrier technology, which has relatively simple manufacturing processes and structures but severe intensity loss and Moiré fringes, which have seriously hampered its development and hindered the clinical application of glasses-free 3D technology. Based on the specific usage scenarios of endoscope systems, we need to consider the viewing distance, viewing angle, and number of viewers of a glasses-free 3D system. On the other hand, due to the special application environments of the system, the system needs to support a wide viewing area and also has strong anti-interference abilities, so as to ensure efficient and real-time image acquisition, processing, and display. Therefore, a mature glasses-free 3D display system must offer reliable stereoscopic display and meanwhile has short learning curve, comfortable operating experience, and low surgical risks. The real glasses-free 3D technology will greatly promote the development of endoscopic surgery.
Design of glasses-free 3D thoracoscopic display systems
The glasses-free 3D display technology refers to a stereoscopic display technology capable of obtaining 3D visual effects without the aid of glasses. The principle is that through the computer image processing technology, two images with parallax are combined into one and displayed on the 3D screen. The lens of the 3D screen will change the traveling direction of the light of the images, the eyes of the viewers can obtain different images, which forms binocular parallax and thus stereoscopic images (4). The lenticular lens technology is also known as biconvex lens or micro-pillar lens 3D technology. A layer of lenticular lenses is added in front of the display screen, so that the image plane of the display screen is located on the focal plane of the lens. Below each cylindrical lens the pixels of the image are divided into several sub-pixels so that the lens can project each sub-pixel in different directions, enabling the left and right eyes to obtain the separated images in the left and right eyes. The pixel images of the left and right eyes on the liquid crystal screen entered the left and right eyes of the observers, respectively, through the refraction effect of the liquid crystal lens, and then form stereoscopic perception through the stereoscopic fusion of the vision center. For liquid crystal displays with different sizes, the relationships between the pixels and the lens, the distance between two pupils, the distance between the stereoscopic viewing area and the display screen, and the area of viewing area must be determined.
Figure 1 shows the viewing area for watching the stereoscopic images, in which Dc represents the closest distance (Dc), optimal distance (Do), and farthest distance (Df). Based on the geometric relationships, the relationships between these parameters can be derived:
Dc = Do (W − K)/W
Df = Do (W + K)/W
where: W and K are the width of the grating and the distance between the pupils of the two eyes, respectively. The distance between two pixels is indicated in Pp. The distance between two pupils is typically 65 mm. PL and R stand for the pitch and radius of curvature (ROC) of the lens, T for the thickness of the liquid crystal lens, and W for the width of the liquid crystal lens.
Pp=[1+(T-R/Do)] PL
Because the optimal viewing distance (Do) of the thoracoscope is 1.8 meters in clinical settings, the parameters of the glasses-free 3D grating film under optimal viewing distance of the thoracoscopic system are obtained [also based on the specification parameter (Pp) of the LCD].
Advantages of glasses-free 3D system
Display brightness
The traditional glasses-assisted 3D thoracoscopic system utilizes the polarized light to achieve stereoscopic display. That is, the glasses worn by the operator has one horizontally polarized lens and one vertically polarized lens, which allow the operator to see the different images brought by horizontally polarized light and vertically polarized light with left and right eyes, respectively, thus forming a 3D display (5). In a glasses-assisted 3D system, only horizontal or vertical light can pass through the lens; as a result, the light passing through glasses is reduced by 50%, and the display brightness remarkably decreases. This is also a problem that has long plagued the traditional thoracoscopic surgical display systems. In order to solve this problem, we adopted the lenticular lens technology in our project protocol, which could effectively protect the brightness from being affected. In principle, the lenticular lens technology adds a layer of lenticular lenses in front of the traditional LCD so that the lens can project each sub-pixel in different directions. The two eyes can see different sub-pixels when watching the LCD from different angles and thus obtain 3D images (6). Because the lenticular lens and the pixel column are not parallel but at a certain angle, each group of sub-pixels can be repeatedly projected onto the viewing zone instead of only projecting a group of parallax images. Therefore, its brightness will not be affected. Furthermore, the lenticular lens does not block the backlight, and therefore the screen brightness can be well guaranteed. A comparative analysis showed that the display brightness of the glasses-free 3D thoracoscopic system is 92% higher than that of the traditional glasses-assisted 3D thoracoscopic system.
Viewing area
How to expand the viewing area of the glasses-free 3D display is a hot research topic in this field. The traditional solution is to use a wider grating of the lens unit and to combine more views to create a 3D effect. Although this method can significantly increase the viewing range, the display resolution and image sharpness remarkably decrease. In our current project, we use tracking and positioning technology to expand the viewing area. This technology can dynamically display the resultant images by feeding back the 3D positions of the viewers in real time and driving change in the results of stereoscopic composition. At present, locating the viewer’s eye position through the face tracking technology is an effective method for obtaining the viewer’s 3D position; however, how to stably detect and track human face through an ordinary camera in real time and convert the 2D coordinates of the human face into 3D coordinates remains an important research topic (7). Due to the sensitivity of human eyes to stereoscopic effect, the tracking/positioning sub-systems need to provide centimeter-level accuracy, along with high requirements on gesture recognition and stability tracking. In our design, an active appearance model (AAM) is used to discriminate the eye positions and various gestures. The 3D coordinate data of the viewer's eyes obtained from the tracking algorithm are fed back to the stereographic image processing chip, which synthesizes the 3D images that can be seen at this specific position in a real-time manner. As a result, the tracked viewers will not be restrained by the viewing area, which effectively expand the viewing area of the glasses-free 3D display.
Anti-interference performance of the system
Since the system is used in a specific setting for thoracoscopic surgery, it has a highly real-time requirement; furthermore, it has high time limits for face tracking, image display, and anti-jamming processing algorithms. Finally, the operating room has a strong lighting environment and is attended by surgeons, assistants, nurses, and other personnel, which increase the complexity of the application environment. In the field of face tracking, how to solve the extreme lighting conditions and address various facial algorithms remain hot research issues. The continuous improvements in camera performance and the use of cameras with special functions (e.g., cameras with infrared component and cameras that can sense depth information) have provided more means for face tracking and will put forward more requirements, which will promote the development of face tracking in the fields of pattern recognition and human-computer interaction. The system should be able to display high-definition and smooth 3D images when the operator walks freely within the designed viewing area when using it. Therefore, the system needs to quickly recognize and position the human eyes and has highly real-time performance. In order to improve the efficiency of human eye recognition, efforts should be made to shorten the time of human eye recognition, so that the system can quickly obtain human eye coordinate information. In our design, we adopt the auxiliary infrared recognition technology. By using the infrared light emitted from an auxiliary device, the system can rapidly position human eyes, which can greatly improve the response time of the system. Due to the active emission mode, the system can automatically recognize the infrared intensity and achieve effective identification, which enhances the processing efficiency of the system and effectively improve its interference performance.
Clinical applications of the glasses-free 3D display system
The glasses-free 3D display system has been applied in our center for two years and has shown good safety and efficiency. With all the advantages of glasses-assisted 3D technology, it avoids the adverse effects of auxiliary glasses. Recently, we retrospectively compared the roles of glasses-free 3D and conventional 2D thoracoscope radical surgery for lung cancer. The intraoperative indicators (mean number of lymph nodes dissected, incidence of massive bleeding, rate of conversion to thoracotomy, and total amount of blood loss), postoperative indicators (volume of chest drainage, drainage days, and complications such as pneumonia and atelectasis), postoperative systemic inflammatory response (e.g., decreased leukocytes and increased body temperature) were not significantly different, demonstrating that glasses-free 3D technology is at least as safe as conventional 2D endoscopic surgery for patients. For operators, the safety concerns of glasses-free 3D are mainly focused on visual acuity. No study has yet compared the impacts of glasses-free 3D and conventional 2D technology on the visual acuity. Although symptoms such as eye fatigue, dizziness, and ghosting had been reported by a small number of operators at the early stage of use, these symptoms quickly disappeared after a short period of adaptation. Post-surgical questionnaire surveys showed most surgeons were more inclined to continue using glasses-free 3D, and no one had complained of a significant decrease in visual acuity. Recent literature found that, compared with traditional 2D technology, the glasses-assisted 3D thoracoscopy did not cause more visual fatigue or cognitive fatigue (8). Therefore, it can be speculated that the impact of the intraoperative application of glasses-free 3D technology on visual acuity would be non-inferior to the 2D technology; however, further clinical trials are needed to validate our assumption. The glasses-free 3D technology has the following advantages:
The glasses-free 3D technology restores the real 3D structures of the chest cavity, making thoracoscopic surgery more accurate and efficient
The 3D high-definition field of view provides a good sense of depth and the 3D structures of lung arteries and veins that can be magnified up to 20× bigger, which allows the surgeons to perform fine operations (such as arterial and venous sheath removal) more confidently. For patients whose anatomical structures are difficult to identify due to enlarged lymph nodes or intrathoracic adhesions, the 3D visual field can clearly display the traveling directions of blood vessels that adhere to the lymph nodes and reduce the incidence of major bleeding events caused by the separation of blood vessels during the operation. With a good sense of depth, the operator can smoothly separate incompletely developed lung fissures and deep adhesions. During the dissection of mediastinal lymph nodes, the glasses-free 3D technology can clearly magnify the deep structures around the lymph nodes and reveal the adjacent structures such as fat, fascia, lymphatic vessels, enabling the operator to carry out accurate dissection and dissociation, which shortens the dissection time and avoids unnecessary surgical injuries. In May 5, 2015, we successfully, and for the first time, performed glasses-free 3D thoracoscope lobectomy in two cases. Since then, we have completed more than 100 cases of glasses-free 3D thoracoscope radical resection of lung cancer. The surgical procedure was smooth in all these cases, without death or severe complications. The glasses-free 3D mediastinal lymph node dissection has significantly shorter operative time than the conventional 2D thoracoscopic surgery (9). In December 2015, we performed a glasses-free 3D thoracic minimally invasive surgery in non-intubated live to the global audiences. Some thoracic patients could be discharged on the day of surgery. It can be foreseen that, with the availability of glasses-free 3D endoscopic equipment, the increase in the anesthetic and surgical techniques, and the acceptance of the concept of enhanced recovery after surgery (ERAS), the endoscopic surgery will be more accurate and efficient, and ambulatory surgery will no longer be out of reach!
The glasses-free 3D technology makes the complicated thoracoscope surgeries safer and more reliable
Fine suturing is a decisive factor in the success of a complex surgery. The glasses-free 3D technology enables the clear display of the 3D morphologies of the inner and outer membranes of trachea and blood vessels and allows the accurate positioning during the insertion and withdrawal of the needle. A good sense of depth effectively overcomes the blind zone of the “coaxially parallel” tubular view in 2D systems and can effectively avoid the distortion of the anastomosis after suturing, making the suturing of trachea and blood vessel under the endoscope more safe and convenient. In a previous experiment performed on porcine trachea in vitro, tissue exposure and needle insertion, withdrawal, changing, and holding can be performed in one smooth motion with the assistance of glasses-free 3D system, along with even and tight sewing pitch and margins. Thus, it may be a promising technique in complex surgeries. With the use of the glasses-free 3D technology, the operator does not need to wear black glasses, thus restoring the “original” surgical environment in the operator's eyes and allowing the operator to communicate with the medical staff in a smooth and orderly manner. Up to now, we have completed many difficult procedures. For instance, we performed glasses-free 3D endoscopic bronchoplasty, arterioplasty, and angioplasty of the superior vena cava for the radical treatment of right middle upper lung cancer (10). Also, we have performed endoscopic carinal resection and reconstruction and endoscopic non-intubated single-port carinal resection. Almost all thoracic procedures such as bullae resection, segmentectomy/lobectomy, tracheal surgery, single-port thymectomy with a subxiphoid approach, and radical surgery of esophageal cancer have been performed by using this technology.
The learning curve of thoracoscope operation becomes shorter by using the glasses-free 3D technology
The traditional 2D technology uses a planar 2D field of vision, and beginners need to spend more time and practice in more cases to improve eye-hand coordination. In addition, the second-generation 3D field of view is relatively dark after its light is filtered by polarizing glasses; therefore, beginners need some time to adapt to the dim light after brightness reduction. The glasses-free 3D system can enlarge and reconstruct the 3D structures of the surgical field under the naked eyes. It will not be affected by the shading of the polarized glasses and can increase the visual field brightness by 50%. It helps beginners to quickly adapt to the visual field and maximize eye-hand coordination, thus reducing the surgical difficulty and shortening the learning curve of endoscopic surgeries. For physicians with experiences in 2D or glasses-assisted 3D endoscopic operations, they often can directly perform these procedures without re-adaption to the glasses-free 3D environment.
Limitations of glasses-free 3D technology
At present, the dual-view glasses-free 3D system is the mainstream product in the market. Medical staff except for the operator still need to wear glasses for viewing. Thus, a wide-angle, high-resolution, multi-view glasses-free 3D system is urgently needed. Although the crosstalk rate has been reduced to 4%, image ghosting still occurs occasionally during surgery. Therefore, the crosstalk rate still needs to be minimized to eliminate ghosting. It is believed that, with the improvement of engineering technology, the glasses-free 3D display system will become more stable. In addition, the dual cameras of the viewing lens have a fixed position and thus the angle of view cannot be changed by rotating the lens. Once there is an object in front of the lens, the camera body has to be rotated to avoid the obstacle. The lens needs to maintain a proper distance from the target area, otherwise the 3D imaging effect will not be good and may cause visual fatigue. For this purpose, the camera holder must be highly professional, with solid knowledge and skills in glasses-free 3D imaging.
With the rapid development of computer processing technology and display technology, the thoracoscopic systems are making enormous strides, which promote the advances in minimally invasive surgery. The glasses-free 3D display technology has been proven to be a promising new 3D technology for thoracoscope surgery. Although its imaging system still needs to be improved and its superiorities over glasses-assisted 3D and traditional 2D technology still need to be validated in further clinical trials, its greatest advantage is to present 3D visual field while getting rid of the shackles of glasses. It can be foreseen that glasses-free 3D display technology will surely become the mainstream mode for future endoscopic surgery. It will dramatically promote the development and application of minimally-invasive endoscopic surgery, with great economic and social benefits.
Acknowledgements
Funding: This study was supported by the Guangdong Provincial Department of Science and Technology 2014 Innovation Project “Research, Development, and Clinical Application of Glasses-free Three-Dimensional (3D) Display Devices for Thoracoscopic Surgery” (grant number: 2014A020215037).
Footnote
Conflicts of Interest: The authors have no conflicts of interest to declare.
References
- Gharagozloo F, Margolis M, Tempesta B, et al. Robot-assisted lobectomy for early-stage lung cancer: report of 100 consecutive cases. Ann Thorac Surg 2009;88:380-4. [Crossref] [PubMed]
- Ninan M, Dylewski MR. Total port-access robot-assisted pulmonary lobectomy without utility thoracotomy. Eur J Cardiothorac Surg 2010;38:231-2. [Crossref] [PubMed]
- Khoshabeh R, Juang J, Talamini MA, et al. Multiview glasses-free 3-D laparoscopy. IEEE Trans Biomed Eng 2012;59:2859-65. [Crossref] [PubMed]
- Wang QH. 3D Display Technology and Device, Science Press, Beijing, 2011.
- Shao HR, Wang NN, He YH, et al. Polarized light imaging and its applications. China Biophysics Academic Conference, 2006.
- Zhang X, Zheng CW, Li N, et al. Liquid crystal materials and 3D display. Chinese Journal of Liquid Crystals and Displays 2012;27:448-55. [Crossref]
- Liang LH, Ai HZ. An algorithm for tracking faces based on face detection. ComEngApp 2001;37:42-5.
- Lin CJ, Cheng CF, Chen HJ, et al. Training Performance of Laparoscopic Surgery in Two- and Three-Dimensional Displays. Surg Innov 2017;24:162-70. [Crossref] [PubMed]
- He JX. Application of glasses-free 3D display system in laparoscopic surgeries. The Journal of Practical Medicine 2017;33:1537-9.
- Shao W, Yin W, Wang W, et al. Glasses-free three-dimensional endoscopic bronchoplasty, arterioplasty, and angioplasty of the superior vena cava for the radical treatment of right middle upper lung cancer. J Thorac Dis 2016;8:608-11. [Crossref] [PubMed]
