Augmented Reality with HoloLens® in Parotid Tumor Surgery: A Prospective Feasibility Study

Surgery of the parotid gland is a major field of otorhinolaryngology, head and neck surgery. However, the anatomy in this region is complex. Important nerves, especially the facial nerve, blood vessels, and muscles have to be treated with care. A balance must be found between function, cosmetics, and an oncologically safe resection. Traditionally, preoperative surgical planning is based on ultrasound, CT, and/or MRI. However, it is still difficult to guarantee sufficient surgical accuracy. In recent years, augmented reality (AR) has spurred research in different surgical specialties to improve accuracy in surgical planning and performance: neurosurgery [1], urology [2], orthopedics [3], and general surgery [4], among others. The peculiarity of this first trial lies in the application to the soft tissue of the head with the direct use during the operations. There are no applications in soft tissue open surgery in otorhinolaryngology, head and neck surgery yet, especially none with head-mounted devices (HMDs) [5]. The previous use of AR HMDs is limited to the use on phantoms, cadavers, or bony structures of patients (Table 1).

In AR, virtual objects are overlaid on a real, physical environment in contrast to virtual reality in which a real environment is replaced by a virtual one [17, 18]. The focus of this pilot study was to gain first trial experience concerning wearable hardware and usability of holograms during real parotid surgeries with special emphasis on preservation of the attention and focus of the surgeon on the patient, improved 3D anatomic evaluation, improved ergonomics viewing MRI images during surgery, and a first-trend estimate of accuracy. We aim to use this system to help surgeons to improve the ability to judge position and orientation by either placing 3D hologram reconstructions over the physical operation field or by loading 2D MRI holograms into the direct visual axis of the surgeon. Thus, this is a promising innovative technology for otorhinolaryngology, head and neck surgery with a “see-through” effect in the operating room.

As an AR system, we chose the Microsoft HoloLens® 1 (Microsoft Corporation, Redmond, WA, USA). It is a wireless system running in Windows 10 on 2-GB RAM and 64-GB storage. In comparison with other devices (such as Google Glass®, USA), a unique feature of the HoloLens® is its automatic interpupillary calibration with gaze tracking and gesture input. The gestures are completely touch free and can be used in sterile environment. The main gestures used are “air tapping” (pinching of thumb and index finger) to select holographic objects and “blooming” (opening hand) to open or close the main menu (Table 2). Regardless of the real spatial environment, 2D and 3D holographic images can be placed anywhere in the user’s visual field. The advantage of this system is the combination of different sensors and cameras in one helmet. The fixed measurement unit, optic sensor placement, and 4 perception cameras are included [6]. The picture quality is 720p. Wearing the 579-g HoloLens® requires a certain amount of habituation. But, similar to a head light, the weight is distributed over the headband over the whole head and is therefore nothing unusual for an ENT surgeon.

An overview of the system is given in Figure 1: to create a holographic image, we extracted DICOM files from MRI. The area that should be shown on the hologram was manually aligned and segmented using the open source Software 3D Slicer [19, 20]. The reconstructed 3D objects are exported as Wavefront OBJ files. Unity3D [21] is used to build the holographic environment containing the desired objects and functionalities of the application. Finally, the virtual scene is deployed onto the HoloLens by Visual Studio [22]. Thus, with HoloLens either a 3D hologram of the segmented structures is visible or the MRI images as 2D slices.

Pre- and intraoperatively, the holographic 3D reconstruction makes it easy to determine the location of the tumor. In 6 patients, the depth of the tumor in the parotid gland and the relationship to neighboring structures were recognizable. Thus, the surgeon could plan the extent and the access of the operation before the actual surgical procedure started and during the procedure before placement of an incision.

To evaluate feasibility during surgery, the first author was trained for 20 min on using the HoloLens as a test person. The training included performing the gestures, operating the menu, viewing the MRI images in different slices, handling the 3D hologram, and switching between the 2D MRI image view and 3D hologram view. The generated hologram can be “touched” by gestures and rotated in all directions of the room or be zoomed in and out. Table 2 shows the individual gestures and their application. The structures that have been segmented for parotid surgery are parotid tumor, parotid gland, mandibula, and masseter muscle. For improved orientation, the segmented structures can be shown and hidden separately (Fig. 2a–d).

In the operation theater, the 3D hologram of the patient’s head and tumor was merged with the registration of the skin surface manually. This allows a position estimate of the tumor border and adjacent structures on the skin for better orientation (Fig. 2b–d).

Wearing the HoloLens in the operating room, the test person was fully dressed up for surgery in all of the 6 tumor operations. A standard set of tasks was performed: opening the main menu, selecting particular applications from the menu, viewing MRI slices, viewing and hiding individual structures of the 3D hologram, zooming of MRI or 3D hologram, and rotating the hologram in all directions of the room. Then, the 3D hologram was merged with the patient’s facial structures. The study person sat next to the surgeon but did not interact with the surgery itself (Fig. 3a, b). During the progress of the operation, the following aspects were checked: match of the hologram and reality in terms of tumor localization and location of neighboring structures, showing and hiding structures and rotating the hologram, switch between reconstructed 3D hologram 2D MRI hologram, scrolling through the MRI slices, zooming of MRI images, and assessing the picture quality and the light intensity. Accuracy of alignment was measured intraoperatively using a sterile electromagnetic navigation pointer (Fiagon AG Medical Technologies, Hennigsdorf, Germany) measuring prespecified landmarks on the 3D holographic structure and real patient’s anatomy. The difference between the points was calculated and displayed using the navigation device.

2D and 3D holograms were successfully created and visualized. Wearing the HoloLens was possible in a completely sterile envirsonment. It fitted comfortably on the head with and without regular glasses under the HoloLens. The weight was distributed around the head and could be carried easily during the entire operation. After the sterile wash-in, no adjustment was necessary, not even after multiple head turns and hand gestures. It ran battery operated throughout the operation. While the physical environment could be seen as normal, the image quality of the holograms was sharp with bright colors. Although subtle visual delays occur when turning the head they did not noticeably affect the workflow and were most noticeable with very rapid movement.

The use of the HoloLens: it offers hands-free access to complex 2D and 3D data for preoperative planning or intraoperative navigation. The actual improvement in using this device is the fact that the surgeon can leave his head facing the patient when viewing MRI images or 3D reconstructions during surgery. In order to view MRI images, neither an additional monitor nor another person is required to open or change the slices. Figure 4 demonstrates intraoperative use of holograms to visualize MRI images. The windows of the HoloLens can be opened, manipulated, and closed by the surgeon in a sterile environment using only hand movements in space. So far, no explicit navigation system for parotid surgery does exist because traditional navigation systems have focused on operations, such as sinus and ear surgery. In contrast to these surgeries, in parotid surgery, there are movable soft tissue structures that previously have made conventional point-to-point navigation difficult. The superimposition of 3D reconstruction with the surgical field can serve as a basic navigation system in parotid surgery. The surgeon sees the anatomical structures in the hologram and thus knows at which anatomical point the surgical instrument is currently located. Showing and hiding structures makes orientation easier. Superficial structures are hidden so that the structures behind them can be seen better. By deliberately hiding the tumor structures, the postoperative appearance of the parotid gland can be anticipated before and during the operation. Figure 2d shows in unprecedented plasticity how much the parotid gland will dent after tumor removal.

The average error in accuracy for each registration based on measurements from the anatomical structures was measured. The mean error of the alignment was 1.3 cm (range 0.58–2.1). The frequency distributions of all of the recorded data are shown in Figure 5. Figure 6 shows that measured error was equally distributed, showing no significant statistical differences among the central segmented structures, parotid and tumor (p = 0.326), and the peripheral structures, mandible and external borders of the head (p = 0.0577). Interestingly, there was a highly significant difference in the accuracy of registration between the central and peripheral structures in general (p = 0.0059). As displayed in Figure 6 and Table 3, the accuracy of the point registration was significantly better in the central structures (parotid gland and tumor) than in the peripheral structures (mandible and outer borders of the head). The parotid gland showed the lowest deviation between the 3D model and patient anatomy, 10.9-mm mean deviation.

To evaluate the surgical time, we compared the recorded surgical preparation time and the incision-suture time with a control group of the same type of parotid surgeries (“extracapsular dissection”) (Fig. 7). The mean surgical preparation time in the AR group was 39.7 min and in the control group 25.7 min (p = 0.001). No significant difference was found in the ultimate incision-suture time (AR group 102.6 vs. 100 min control group, p = 0.439).

Interestingly, no saliva fistula as a complication occurred in the AR group. But, in the control group, this reversible complication occurred in 3.7%. A temporary partial facial nerve palsy appeared equally often in both groups (p = 0.5). No permanent facial nerve complication was seen in both groups.

Until now, no clinical studies have been published using an HMD in soft tissue procedures of the head and neck area, such as parotid surgery. Since HMDs have been shown to be effective in other types of surgery (e.g., urology and neurosurgery), authors such as for example, Yoon et al. [5], highly recommend future research to evaluate and adopt HMDs for otolaryngology. This pilot study shows the clinical feasibility of a head-mounted AR device for preoperative planning and multiple intraoperative benefits using 3D hologram reconstructions in parotid tumor surgery. The superposition of 3D reconstruction and patient’s face showed that tumor alignment and the alignment of the neighboring structures is possible. We have implemented gestures that allow interaction between virtual and real world by allowing virtual objects to be displayed and hidden. This makes orientation in the surgical site easier, especially if several masses are distributed in the parotid gland. 2D and 3D information is available right in front of the surgeon’s eyes. In 2D mode, MRI images can be viewed via the HoloLens. This is a benefit in terms of preservation of attention and focus on the patient. It also improves ergonomics because the surgeons can keep their focus on the patient’s head, instead of having to look back and forth between patient and MRI screen. This principle has already been noticed in the use of AR devices for navigation systems in neurosurgery [23]. There is evidence that a disrupted visual-motor axis during surgery can lead to a plethora of problems including declined ergonomics and surgical performance, spatial disorientation, and increased risk of iatrogenic injuries [24]. Due to the built-in camera in the HoloLens with integrated computer, no other devices such as camera and monitors have to be set up in the operating room. This saves space and improves the workflow. Another advantage besides improved ergonomics is that the mental stress on the surgeon is also reduced because he can very quickly call up all image information during the surgery itself via the HoloLens, without leaving the operating field or having to ask other people for information. As a result, the operation continues without interruption and can be ended faster. To the best of our knowledge, there have been no studies on wearable AR devices with 3D virtual projections for parotid tumor surgery to this date. Table 1 shows the current literature on AR with HMD or endoscope application in otolaryngology. The application in the real operating room setting is not established. This and the application of AR in soft tissue surgery is a special feature of this study. Until now, there are also no special navigation systems for these operations. The presented AR system can act as a navigation-like tool, for improved anatomical orientation as used in neurosurgery [23, 25]. Using the HoloLens as a navigation-like tool can reduce complications. The overlay of the 3D hologram with the surgical field is extremely helpful in finding the mass and opening the parotid capsule in the right place. In the surgical technique of “extracapsular dissection,” this small and precise incision is aimed for. The parotid capsule is only opened in the area where exactly the mass is in the gland. This means that less glandular tissue is exposed, and complications caused by exposed glandular tissue such as salivary fistula are potentially reduced. Salivary fistula is a common complication of “extracapsular dissection.” Mantsopolous et al. [26] describe the rate of salivary fistula after extracapsular dissection to be 10%. None of our AR patients developed a salivary fistula. But, the rate in the control group was 3.7%. With the introduced HMD AR system, the position of the mass inside the gland can be determined more precisely during the operation. This may lead to less exposed glandular tissue from which saliva can flow into the wound. Since there are no sufficient ways to image the course of the facial nerve in the parotid gland in MRI [27], at the moment it is not possible to visualize the nerve inside the gland with AR. Facial nerve irritation with temporary palsy occurred equally often in study patients with AR and in control patients without AR. Nevertheless, the use of AR could help secondarily to prevent iatrogenic facial palsy by making the extracapsular technique more often possible to perform. Compared to other parotidectomy techniques (lateral or total parotidectomy), this surgical technique has a lower rate of facial nerve lesions [28, 29].

The surgical preparation time was longer than in the control group. There are 2 reasons for this. First, the first study measurement time is already integrated into the recorded preparation time, in which the correspondence of the anatomical outer boundaries with the hologram was measured. This took 5–10 min each, which had nothing to do with the ultimate preparation for the surgery, but took place during the time after the sterile drape and before the incision. Thus, it was automatically recorded by the hospital surgery time measuring software into the surgical preparation time and extended it due to study measures. Second, the lack of routine in deploying the new technology prolonged the preparation time. That also has to do with manual alignment. For this pilot study, we used a completely manual alignment, which must be carried out slowly and carefully so that there is an exact superposition of the 3D hologram and surgical field. Regarding the question of time in general, the quick access to MRI images can make up some time during the surgery that would otherwise be used to open or scroll through the MRI images on a conventional monitor.

The registration error was found to be low at central structures, but it was still relatively high when considering central and peripheral structures together (median 1.6 cm, Fig. 5). On the one hand, this is due to the fact that the registration in soft tissue is subject to self-mobility in comparison with fixed bony structures, and on the other hand, rotation inaccuracies can occur with manual registration. Our results suggest that registration accuracy decreases with increasing distance from the center of the 3D hologram (Fig. 6; Table 3). The more central the structures of the 3D hologram are, the less is the effect of rotational inaccuracies. The best correlation of the 3D holographic picture and real anatomy was seen at the parotid gland (Fig. 2c, d, 6; Table 3). Further investigations using different methods must be carried out to confirm this. Little is currently known about the possibilities for numerical registration accuracy of the AR overlay compared to the real situs. We applied a point-to-point comparison between the real situs and the superimposed 3D model. In neurosurgery, Incekara et al. [23] manually outlined the HoloLens projections compared to a conventional neuronavigation (Brainlab, Feldkirchen, Germany) of tumor borders. As an outcome for accuracy, they used the maximum distance (cm) between the center of the tumor and the tumor borders compared with the true tumor borders of the patient. A semiautomatic registration with fixed points was presented by Mitsuno et al. [8]. Here, 3 fixed points in the hologram are matched with corresponding points on the body surface.

The 3D objects were calculated from the segmentation of the DICOM data. Wavefront OBJ format_I was chosen to be the interface format between the segmentation software Slicer 3D and the application development software Unity3D. The OBJ format has the advantage over the more recent FBX format [30] that it is supported by both programs. The data structure is open and well established. The STL format contains partially similar geometrical information to the OBJ format and it is also open, but it cannot map texture of an object, and it is supported by Unity3D.

Besides HoloLens, there are other HMDs on the market, such as Google Glass or Oculus Rift. All HMDs have evolved from being heavy, obstructive, and wired devices to become light, see-through, and wireless. The HoloLens offers more immersive technology compared to previous HMD generations and enables the user to visualize multiple holograms simultaneously, allowing integration of other important medical information [31] such as viewing MRI slices. Tepper et al. [9] explored Google Glass in maxillofacial surgery and support its promise in surgical use but found it significantly limited by its inability to access critical data in a hands-free manner. Oculus Rift has been used preoperatively in combination with standard imaging modalities to familiarize junior surgeons with the anatomy of the patient. However, this device is handicapped by exclusion of information from the actual environment, which has important implications for use in the operating room [32]. HoloLens is ideal for intraoperative use with its hands-free operation to maintain sterility, robust battery life, a comfortable form factor, and the ability to visualize holograms in a “real” environment, which is not offered by Google Glass or Oculus Rift [9]. These HMDs only show the virtual reality completely replacing the physical environment. AR merges virtual reality with the real environment and is often referred to as “mixed reality.” Therefore, there is virtual and real information of the patient, making this method promising for navigation.

The presented AR application was tested in parotid surgery, as the first HMD soft tissue application in otorhinolaryngology. Regarding augmented scene parameters, all virtual image sources apply, and visualization occurs by holographic images or overlays. Benefits of the method are (1) improved workflow and ergonomics by keeping the focus on the patient when viewing MRI slices during surgery; (2) “extracapsular dissection,” which has fewer complications than other parotidectomy techniques, can be used more easily and more often; and (3) if the extracapsular technique is used, orientation within the gland is improved by additional 3D information. Therefore, less glandular tissue must be exposed which reduces the rate of saliva fistulas.

Nevertheless, our results suggest that manual alignment is time consuming, and it may not seem to be sufficient to reach high accuracies, especially not in the periphery of the 3D hologram. Further intervention with high numbers of patients is necessary to improve the accuracy and clinical applicability of this HMD. Thus, we continue to develop the system by refining skin surface registration and building automatic superposition with the 3D hologram. However, the shown HoloLens application represents the first feasible system for parotid surgery without compromising sterility and supporting orientation and workflow. It can serve as the basis for further measurements.

There are limitations of the current study and the -HoloLens system. The rechargeable battery can support 2–3 h of active use. In some circumstances, this might be too short for more complex interventions or revision surgeries on the parotid gland. In this case, the HoloLens might have to be exchanged for another one, so that the battery can charge. The tracking of the image when the head is turned is a bit slow and takes some adaption. A further limitation of the hardware is that no magnification has been integrated yet, which can be helpful in parotid surgery. In this study, feasibility was successfully assessed, but for the purpose of more quantitative outcome measures, the system needs to be applied to more cases. Unfortunately, we were unable to determine the actual operating costs affecting the surgery, as it is currently still a study project.

We would like to thank the Fraunhofer Institute in Mannheim and DFC-SYSTEMS GmbH, Munich, for making the HoloLens available.

The research was conducted ethically in accordance with the http://www.wma.net/en/30publications/10policies/b3/index.html. Subjects have given their written informed consent, and the study protocol was approved by the institute’s committee of ethics (#2019-739N). The setup of the current study did not impose any additional risks to patients or surgeons because they were not subject to any specific procedure, nor were they required to follow any new rules of behavior. Diagnostics and surgery were performed without any deviation from the standard procedure.

The authors have no conflicts of interest to declare.

The authors did not receive any funding.

Claudia Scherl, Johanna Stratemeier, Nicole Rotter, Jürgen Hesser, Stefan O. Schoenberg, Jérôme Servais, David Männle, and Anne Lammert have substantially contributed to the conception and design of the work with acquisition, analysis, and interpretation of data for the work. Claudia Scherl and Johanna Stratemeier were drafting the work. Nicole Rotter, Jürgen Hesser, Stefan O. Schoenberg, Jérôme Servais, David Männle, and Anne Lammert were revising it critically for important intellectual content. All authors gave final approval of the version to be published and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.