Introduction: A 2½ D point cloud registration method was developed to generate digital twins of different tissue shapes and resection cavities by applying a machine learning (ML) approach. This demonstrates the feasibility of quantifying soft tissue shifts. Methods: An ML model was trained using simulated surface scan data obtained from tumor resections in a pig head cadaver model. It hereby uses 438 2½ D scans of the tissue surface. Tissue shift was induced by a temperature change from 7.91 ± 4.1°C to 36.37 ± 1.28°C. Results: Digital twins were generated from various branched and compact resection cavities (RCs) and cut tissues (CT). A temperature increase induced a tissue shift with a significant volume increase of 6 mL and 2 mL in branched and compact RCs, respectively (p = 0.0443; 0.0157). The volumes of branched and compact CT were decreased by 3 and 4 mL (p < 0.001). In the warm state, RC and CT no longer fit together because of the significant tissue deformation. Although not significant, the compact RC showed a greater tissue deformation of 1 μL than the branched RC with 0.5 μL induced by the temperature change (p = 0.7874). The branched and compact CT forms responded almost equally to changes in temperature (p = 0.1461). Conclusions: The simulation experiment of induced soft tissue deformation using digital twins based on 2½ D point cloud models proved that our method helps to quantify shape-dependent tissue shifts.

Point clouds allow the acquisition and display of 2½ D images. Point clouds are characterized by their ease of use, strong adaptability, low computational complexity, and high detail reproduction ability. Point clouds are the natural representation of surface scanning techniques, such as laser scanning or stereo. Therefore, changes in surface geometry can be directly captured using point cloud registration methods. In particular, tissue surface can be measured and represented as point clouds, and appropriate registration approaches can be used to measure tissue deformation [1].

Tissue deformation, also referred to as tissue shift, is a highly relevant problem in soft-tissue surgery. This causes landmarks to be displaced, making orientation in soft tissues challenging. Depending on its consistency and shape, tissue deforms after wound opening owing to loss of tension, patient positioning, or tissue extraction. So far, this effect has been extensively described in neurosurgical literature [2, 3]. Tissue displacement is also a problem in head and neck surgery, especially during tumor resection, where many critical structures are located in close proximity and potentially require protection.

After tumor surgery, the pathological TNM classification determines the course of therapy and prognosis. To determine the pathological T stage, the tumor must be examined by a pathologist, which is a difficult task in the presence of malformation. Different tumor formations and tissue shifts also pose challenges for pathologists and surgeons in the management of frozen sections.

The main goal of this study was to develop an ML-based strategy that facilitates the quantification of different tumor shapes and determines their changes during tissue shift using point cloud registration and digital twins. To the best of our knowledge, this is the first approach to handle shape-dependent tissue shifts.

Animal Cadavers and 2½ D Scans

This experimental study was performed using 45 pig head cadavers (Schradi Frischfleisch GmbH, Mannheim, Germany). The study was approved by the Mannheim Veterinary Office (DE 08 222 1019 21). Thus, large amounts of data can be generated to train AI. Real tumor material is only available to a limited extent and cannot be reproduced. Therefore, we conducted a cadaver study.

After sagittal dissection of the pig heads, immediate processing was done at 7.91 ± 4.1°C or storage at −18°C. To simulate tumor resection, surgical drapes were placed in the cheek region, and tissue blocks in the cheek area were dissected. Identification marks were drawn on the drapes to mark the region as “region of interest” so that the 3D cameras can identify this area. In order to create reproducible and measurable tissue shape variants, the following shapes were cut out: “L,” “T,” “I,” “triangle,” “circle,” “square,” and “rectangle” (Fig. 1). The cut shapes (CT) and their corresponding resection cavities (RCs) were scanned using 3D cameras. Artificial tissue deformation was induced by controlled heating of CT and RC in a heating cabinet (Binder GmbH, FD-53) for 10–12 h from 7.91 ± 4.1°C to 36.37 ± 1.28°C. The core temperatures were measured using a piercing probe. After warming, the scans were performed again. All raw data scans were obtained with a head-mounted display HoloLens 2© (Microsoft HoloLens 2©, Microsoft Corporation, Redmond, WA, USA) and a 3D-object scanner (Artec 3D, Luxembourg). Both have already been proven to be sufficient for the clinical and experimental use of head and neck surgery [4‒6].

Fig. 1.

Head-mounted display images of RCs and cut tissue piece. Arrows and lines point to “cranial,” “caudal,” “rostral,” and “occipital” directions to indicate the region of interest to the 3D cameras and photogrammetry. a T-shape as an example of branched shapes. a1 cold RC. a2 warm RC. a3 cold CT. a4 warm CT. b Rectangular shape as an example of compact shapes. b1 cold RC. b2 warm RC. b3 cold CT. b4 warm CT. Heating leads to the enlargement of RC (a2, b2), particularly visible in the circled area of the compact RCs in b1 and b2. CT is subject to shrinkage and flattening due to heating (a4, b4). RCs, resection cavities; CT, cut tissue piece. Tissue temperature: warm = 7.91 ± 4.1°C, cold = 36.37 ± 1.28°C.

Fig. 1.

Head-mounted display images of RCs and cut tissue piece. Arrows and lines point to “cranial,” “caudal,” “rostral,” and “occipital” directions to indicate the region of interest to the 3D cameras and photogrammetry. a T-shape as an example of branched shapes. a1 cold RC. a2 warm RC. a3 cold CT. a4 warm CT. b Rectangular shape as an example of compact shapes. b1 cold RC. b2 warm RC. b3 cold CT. b4 warm CT. Heating leads to the enlargement of RC (a2, b2), particularly visible in the circled area of the compact RCs in b1 and b2. CT is subject to shrinkage and flattening due to heating (a4, b4). RCs, resection cavities; CT, cut tissue piece. Tissue temperature: warm = 7.91 ± 4.1°C, cold = 36.37 ± 1.28°C.

Close modal

Data Generation

After the scans, images for photogrammetry were taken using the HoloLens. Meshroom software (Version 2021.1.0, Windows 10, Python 3.7.4) was used as a GUI for the Alice Vision Framework on a workstation with 16 cores (AMD Ryzen 9 5950X), 64 GB RAM, and a single Nvidia 3080 Ti GPU. MeshLab 64 bit, v2021.07, and Blender 2.93.4 meshes were generated in Artec Studio 14 Professional. Individual 2D images from various angles are taken to form a 2½ D mesh, which is recognized and compared by Meshroom and is thus able to determine the different camera positions.

Postprocessing

Postprocessing is necessary to improve the quality of the 2½ D objects using MeshLab software. For example, if there are holes on the surface, the “Close-Holes” function is applied. The total number of vertices for the entire mesh is set to 300,000. MeshResampling node is used and the “Simplification factor” is set to 1.0 because the surface of the object is not captured evenly resulting in irregular distribution of the vertices on the 2½ D figures. Artec Studio Professional 14 software was applied to develop the 2½ D meshes from the scans with Artec Eva, which also has a hole-closing function and a region of interest segmentation function. The Meshroom provides 2½ D figures of the HoloLens in an OBJ format, which does not provide metric information of length and 2½ D object volume. To obtain a calculable volume of CT and RC, an adaption to the real size must be created. Therefore, a scaling factor must be calculated using the metric information of Artec meshes. The length of the cranial mark (Fig. 1) was determined as “ltarget” in millimeters. The mesh for the HoloLens is measured in the same way and is shown as “lsource,” which is dimensionless. The scaling factor f is calculated using the following equation:
f=ltargetlsource
(1)

Segmentation and Volume

Segmentation was performed using the MeshLab. The region of interest has to be cut out from the rest of the mesh using the “Draw Polyline” tool, leaving only the Sh and the RC for subsequent evaluation. The volumes were measured in mm3 and calculated using the MeshLab software.

Statistics

To compare differences in volume depending on temperature and shape, the Wilcoxon signed-rank test was used. This study examines whether the central tendencies of two dependent paired samples differ. The median, standard deviation (SD), minimum, maximum, 95% confidence interval (CI), and p value were calculated for exact correlations. Differences were considered statistically significant at a p value of 0.05. GraphPad PRISM, Version 9, 2020, was used for all analyses.

In total, 438 scans of cold and warm tissues derived from 90 halved pig heads were obtained. In the 38 datasets, volume determination was impossible owing to quality issues. Therefore, these data were excluded from the present evaluation, leaving 400 datasets for analysis. Artec Eva scans were used to evaluate the shape dependence. The shapes are divided into branched and compact configurations. For the analysis of the branched shapes, L-shape, T-shape, and triangular shapes were integrated. Correspondingly, circular, square, rectangular, and I-shapes were considered for the analysis of compact shapes (Fig. 1). The shape-dependent stratification of RC and CT in the cold and warm states with respect to their respective volumes is shown in Figure 2.

Fig. 2.

Volume as a function of temperature for cold and warm scans. a Branched tissue shapes. b Compact tissue shapes. RC, resection cavity; CT, cut tissue shape; cold scan, 7.91 ± 4.1°C; warm scan, 36.37 ± 1.28°C. Significance was determined with the Wilcoxon signed-rank test: ***p ≤ 0.0001, highly significant; **p ≤ 0.01, very significant; *p ≤ 0.05, significant.

Fig. 2.

Volume as a function of temperature for cold and warm scans. a Branched tissue shapes. b Compact tissue shapes. RC, resection cavity; CT, cut tissue shape; cold scan, 7.91 ± 4.1°C; warm scan, 36.37 ± 1.28°C. Significance was determined with the Wilcoxon signed-rank test: ***p ≤ 0.0001, highly significant; **p ≤ 0.01, very significant; *p ≤ 0.05, significant.

Close modal

Volume Depending on the Temperature

For branched shapes (Fig. 1a, 2a), the median and SD of the RC volumes were 42 ± 22 mL in the cold state (range: 24–118 mL, 95% CI: 34–53 mL). The median volume was 48 ± 23 mL after heating (range: 22–118 mL, 95% CI: 30–54 mL). Temperature change increased the volume of the RC by 6 mL from 42 μL to 48 μL (p = 0.0443).

For the volumes of branched CT, highly significant changes were observed after the temperature change. Therefore, the volume of the CT in the cold state was 42 ± 19 mL (range: 22–92 mL, 95% CI: 28–50 mL) compared to the warm state 40 ± 18 mL (range: 20–86 mL, 95% CI: 25–44 mL), resulting in a decrease from 42 mL to 40 mL by a total of 2.0 mL (p < 0.0001). In the warm state, the volumes of RC (48 mL) and CT (40 mL) differed significantly, so they no longer fit together (p = 0.001).

In the analysis of compact forms (Fig. 1b, 2b), significant changes due to temperature alterations are observed. The volume was 31 mL in the cold RC group (31 ± 11 mL, range: 15–65 mL, 95% CI: 26–38 mL). These amounts were lower than the volume of 34 mL in the warm state (33 ± 13 mL, range: 13–62 mL, 95% CI: 27–46 mL) by 3 mL (p = 0.0157).

A comparison of the volumes of the compact CT shows that there is a decrease in the volume after an increase in temperature. The median volumes and SD of cold (range: 16–49 mL, 95% CI: 25–35 mL) and warm (range: 14–43 mL, 95% CI: 22–32 mL) measurements were 28 ± 9 mL and 24 ± 8 mL. The volume was decreased by 4 mL from 28 mL to 24 mL (p < 0.0001). In the warm state, RC (34 mL) and CT (24 mL) were no longer accurately fitted with a volume difference of 9 mL (p < 0.0001).

Volume Change ΔV as a Function of Tissue Shape

As shown in Table 1, the volume changes of the branched and compact RC and CT were checked for differences based on the total number of branched and compact geometric shapes in each group. Although not significant, the compact RC showed a greater tissue deformation of 1 mL than the branched RC (0.5 mL induced by the temperature change). The branched and compact CT forms responded almost equally to changes in temperature (p = 0.1461 trend).

Table 1.

Amount of volume change ΔV induced by altering tissue temperature

Tissue typeΔVMedian VSDRange95% CIp value
RC branched 0.5 1.5 5.2 25.6 0.17–4.73  
RC compact 1.0 5.8 25.1 0.1–7 0.7874 
CT branched 2.4 2.5 10 2.3–5.2  
CT compact 2.7 2.7 1.6 2.1–3.7 0.1461 
Tissue typeΔVMedian VSDRange95% CIp value
RC branched 0.5 1.5 5.2 25.6 0.17–4.73  
RC compact 1.0 5.8 25.1 0.1–7 0.7874 
CT branched 2.4 2.5 10 2.3–5.2  
CT compact 2.7 2.7 1.6 2.1–3.7 0.1461 

All data in mL; RC, resection cavity; CT, cut tissue.

Using this method, it is possible to induce and quantify tissue deformation, also referred to as the tissue shift. Tissue shift occurs primarily in soft tissues and represents a problem in soft-tissue surgery. Thus far, it has not been possible to satisfactorily register soft-tissue shifts. There are different approaches, all of which are based on repeated intraoperative imaging [7‒9]. With this novel ML-based point-cloud method, it is possible to register deformations in real time via point-cloud registration. Hence, tissue shifts can be precisely registered without imaging (CT, MRI, or sonography). This saves time and effort and reduces radiation exposure.

Point clouds are an elegant method to represent a 3D object. In addition to fields such as robotics [10] and autonomous driving [11], point clouds have become increasingly important in medicine [12‒14]. We chose this method because it is particularly suitable for capturing different 2½ D shapes [15]. We confirmed that the digital twins generated by point clouds can be used to register a wide variety of geometric tissue shapes. It does not matter whether the shapes are highly branched or compact. All shapes responded to temperature changes, and it was possible to register this change regardless of the shape (Table 1). Although real tumor forms are individually misshapen, in this study, we deliberately used geometric shapes to make the process experimentally reproducible. The point-cloud registration technique can be used for all tissue formations. This is important because very different tumor shapes can occur, especially during head and neck surgery.

In all cases, a significant tissue shift was caused by a change in temperature (Fig. 2). Tissue tension is lost and dehydration occurs when the tissue temperature raises, especially the compact RC gape. This process simulates the tissue behavior during head and neck tumor resection. Once a tumor is removed, the tension of the surrounding tissue leads to enlargement of the defect. This effect was most pronounced after heating compact RC and CT (Fig. 2b). It also occurs in branched forms but is not as distinct because of the longer resection borders (Fig. 2a). The warm CT undergoes significant deformation such that it no longer fits into the corresponding RC. This phenomenon is clinically well known and highly relevant in frozen section management. The pathologist receives a tumor with a soft and deformed consistency. The dimensions of this tissue no longer corresponded to the equivalent points in the RC. If a pathologist finds tumor cells at the resection margins of the tumor, re-resection must be performed. Owing to tissue deformation, it is extremely difficult to find the right place in the RC for re-resection. The target deviation for subsequent re-resection can be up to 1 cm [16]. As a result, in 78% of all re-resection samples, no residual tumor was found, meaning that the corresponding site in the tumor bed for the positive or close margins was incorrectly localized, and tumor cells might remain in the patient [17]. The point cloud registration and the creation of digital twins could help to assign corresponding points on the removed piece of tissue to the correct point in the RC because we can use the technology to measure the tissue shift depending on the tissue shape.

Tumor size and extent determine the classification of tumors using the TNM system [18]. As the prognosis is derived from this, it is of great importance to make a valid pathological statement. In the case of misshapen and branched tumors, it is difficult to determine their actual size. Here, our point-cloud-based AI method can help correctly classify the actual tumor size when determining the volume.

To use point clouds and digital twins in the future in real head and neck tumor resections with frozen sections, further work is required to train the AI and ensure accuracy. The simulation experiment of induced soft tissue deformation using digital twins based on 2½ D point cloud models showed that our method is effective and correct.

In this study, we developed a method with which tissue shift can be registered in a large number of different tissue shapes. It is also possible to simulate the clinical phenomenon in which tumors and RCs change during the resection process so much that they no longer fit together. This effect is stronger in compact tissue forms than in branched forms, which are characterized by tissue extensions. Here, our method of AI-based point cloud registration can help measure tissue shift, which is useful for solving deformation problems during head and neck surgery in the future.

We are grateful to Johann Kern and Petra Prohaska for providing us with laboratory space and heating cabinets for the cadaver experiments. We would like to thank Dennis Feiler and Holger Ladewig for technical support.

The study was approved by the Mannheim Veterinary Office (DE 08 222 1019 21).

The authors have no conflicts of interest to declare.

The study was funded by the German Federal Ministry of Economic Affairs and Climate Action, Central Innovation Programme for small- and medium-sized enterprises (SMEs) (funding number: KK5044704CS0).

All authors substantially contributed to the manuscript and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work were appropriately investigated and resolved. Sara Monji-Azad: conception and design of the study, acquisition of data, analysis, and drafting the article. David Männle: conception and design of the study and acquisition, analysis, and interpretation of data. Jürgen Hesser: conception and design of the study and drafting the article. Jan Pohlmann: acquisition, analysis, and interpretation of data. Nicole Rotter: drafting the article and revising it critically for important intellectual content. Annette Affolter: interpretation of data and revising the manuscript critically for important intellectual content. Cleo Aron Weis: conception and design of the study and revising the manuscript critically for important intellectual content. Sonja Ludwig: acquisition of data and analysis. Claudia Scherl: conception and design of the study; acquisition, analysis, and interpretation of data; and drafting the article or revising it critically for important intellectual content. All authors approved the final version to be published.

All the datasets on which the conclusions of the manuscript are based are available at a data repository: DOI 10.5281/zenodo.8172219.

1.
Cheng
Q
,
Sun
P
,
Yang
C
,
Yang
Y
,
Liu
PX
.
A morphing-Based 3D point cloud reconstruction framework for medical image processing
.
Comput Methods Programs Biomed
.
2020
;
193
:
105495
.
2.
Miyagi
Y
,
Shima
F
,
Sasaki
T
.
Brain shift: an error factor during implantation of deep brain stimulation electrodes
.
J Neurosurg
.
2007
;
107
(
5
):
989
97
.
3.
Reinertsen
I
,
Lindseth
F
,
Askeland
C
,
Iversen
DH
,
Unsgård
G
.
Intra-operative correction of brain-shift
.
Acta Neurochir
.
2014
;
156
(
7
):
1301
10
.
4.
Scherl
C
,
Männle
D
,
Rotter
N
,
Hesser
J
,
Stallkamp
J
,
Balkenhol
T
et al
.
Augmented reality during parotid surgery: real-life evaluation of voice control of a head mounted display
.
Eur Arch Oto-Rhino-Laryngol
.
2023
;
280
(
4
):
2043
9
.
5.
Scherl
C
,
Stratemeier
J
,
Karle
C
,
Rotter
N
,
Hesser
J
,
Huber
L
et al
.
Augmented reality with HoloLens in parotid surgery: how to assess and to improve accuracy
.
Eur Arch Oto-Rhino-Laryngol
.
2021
;
278
(
7
):
2473
83
.
6.
Scherl
C
,
Stratemeier
J
,
Rotter
N
,
Hesser
J
,
Schönberg
SO
,
Servais
JJ
et al
.
Augmented reality with HoloLens® in parotid tumor surgery: a prospective feasibility study
.
ORL J Otorhinolaryngol Relat Spec
.
2021
;
83
(
6
):
439
48
.
7.
Ivan
ME
,
Yarlagadda
J
,
Saxena
AP
,
Martin
AJ
,
Starr
PA
,
Sootsman
WK
et al
.
Brain shift during bur hole-based procedures using interventional MRI
.
J Neurosurg
.
2014
;
121
(
1
):
149
60
.
8.
Miga
MI
,
Sun
K
,
Chen
I
,
Clements
LW
,
Pheiffer
TS
,
Simpson
AL
et al
.
Clinical evaluation of a model-updated image-guidance approach to brain shift compensation: experience in 16 cases
.
Int J Comput Assist Radiol Surg
.
2016
;
11
(
8
):
1467
74
.
9.
Mohammadi
A
,
Ahmadian
A
,
Azar
AD
,
Sheykh
AD
,
Amiri
F
,
Alirezaie
J
.
Estimation of intraoperative brain shift by combination of stereovision and Doppler ultrasound: phantom and animal model study
.
Int J Comput Assist Radiol Surg
.
2015
;
10
(
11
):
1753
64
.
10.
IMLS-SLAM
Deschaud
J-E
.
Scan-to-model matching based on 3D data
.
IEEE international conference on robotics and automation (ICRA)
.
IEEE
.
2018
.
11.
Li
Y
,
Ma
L
,
Zhong
Z
,
Liu
F
,
Chapman
MA
,
Cao
D
et al
.
Deep learning for lidar point clouds in autonomous driving: a review
.
IEEE Trans Neural Netw Learn Syst
.
2021
;
32
(
8
):
3412
32
.
12.
Li
J
,
Deng
Z
,
Shen
N
,
He
Z
,
Feng
L
,
Li
Y
et al
.
A fully automatic surgical registration method for percutaneous abdominal puncture surgical navigation
.
Comput Biol Med
.
2021
;
136
:
104663
.
13.
Mulay Av
CV
,
Mulay
A
,
Ahuja
B
.
Patient specific bone modeling for minimum invasive spine surgery
.
J Spine
.
2015
;
04
(
04
):
2
.
14.
Wei
S
,
Kam
M
,
Wang
Y
,
Opfermann
JD
,
Saeidi
H
,
Hsieh
MH
et al
.
Deep point cloud landmark localization for fringe projection profilometry
.
J Opt Soc Am Opt Image Sci Vis
.
2022
;
39
(
4
):
655
61
.
15.
Yoo
H
,
Jun
K
.
Point cloud classification by domain adaptation using recycling max pooling and cutting plane identification
.
Sensors
.
2023
;
23
(
3
):
1177
.
16.
Kerawala
CJ
,
Ong
TK
.
Relocating the site of frozen sections: is there room for improvement
.
Head Neck
.
2001
;
23
(
3
):
230
2
.
17.
Chang
AM
,
Kim
SW
,
Duvvuri
U
,
Johnson
JT
,
Myers
EN
,
Ferris
RL
et al
.
Early squamous cell carcinoma of the oral tongue: comparing margins obtained from the glossectomy specimen to margins from the tumor bed
.
Oral Oncol
.
2013
;
49
(
11
):
1077
82
.
18.
Brierley
JD
,
Gospodarowicz
MK
,
Wittekind
C
TNM classification of malignant tumours
.
Wiley
.
2017
.