Abstract
Background: Multiple types of surgical cameras are used in modern surgical practice and provide a rich visual signal that is used by surgeons to visualize the clinical site and make clinical decisions. This signal can also be used by artificial intelligence (AI) methods to provide support in identifying instruments, structures, or activities both in real-time during procedures and postoperatively for analytics and understanding of surgical processes. Summary: In this paper, we provide a succinct perspective on the use of AI and especially computer vision to power solutions for the surgical operating room (OR). The synergy between data availability and technical advances in computational power and AI methodology has led to rapid developments in the field and promising advances. Key Messages: With the increasing availability of surgical video sources and the convergence of technologies around video storage, processing, and understanding, we believe clinical solutions and products leveraging vision are going to become an important component of modern surgical capabilities. However, both technical and clinical challenges remain to be overcome to efficiently make use of vision-based approaches into the clinic.
References
1.
Maier-Hein
L
, Vedula
SS
, Speidel
S
, Navab
N
, Kikinis
R
, Park
A
, et al. Surgical data science for next-generation interventions
. Nat Biomed Eng
. 2017
Sep
;1
(9
):691
–6
.
[PubMed]
2157-846X2.
3.
Jung
JJ
, Jüni
P
, Lebovic
G
, Grantcharov
T
. First-year Analysis of the Operating Room Black Box Study
. Ann Surg
. 2020
Jan
;271
(1
):122
–7
.
[PubMed]
0003-49324.
Prince
SJ
. Computer Vision: Models Learning and Inference
. Cambridge University Press
; 2012
. 5.
Bernal
J
, Tajkbaksh
N
, Sánchez
FJ
, Matuszewski
BJ
, Hao Chen
, Lequan Yu
, et al. Comparative validation of polyp detection methods in video colonoscopy: results from the MICCAI 2015 endoscopic vision challenge
. IEEE Trans Med Imaging
. 2017
Jun
;36
(6
):1231
–49
.
[PubMed]
0278-00626.
Ahmidi
N
, Tao
L
, Sefati
S
, Gao
Y
, Lea
C
, Haro
BB
, et al. A Dataset and Benchmarks for Segmentation and Recognition of Gestures in Robotic Surgery
. IEEE Trans Biomed Eng
. 2017
Sep
;64
(9
):2025
–41
.
[PubMed]
0018-92947.
Stauder
R
, Ostler
D
, Kranzfelder
M
, Koller
S
, Feußner
H
, Navab
N
. The TUM LapChole dataset for the M2CAI 2016 workflow challenge.
2016
; https://arxiv.org/abs/1610.092788.
Flouty
E
, Kadkhodamohammadi
A
, Luengo
I
, Fuentes-Hurtado
F
, Taleb
H
, Barbarisi
S
, et al. CaDIS: Cataract dataset for image segmentation.
2019
; https://arxiv.org/abs/1906.115869.
Gholinejad
M
, J Loeve
A
, Dankelman
J
. Surgical process modelling strategies: which method to choose for determining workflow?
Minim Invasive Ther Allied Technol
. 2019
Apr
;28
(2
):91
–104
.
[PubMed]
1364-570610.
Gill
S
, Stetler
JL
, Patel
A
, Shaffer
VO
, Srinivasan
J
, Staley
C
, et al. Transanal Minimally Invasive Surgery (TAMIS): Standardizing a Reproducible Procedure
. J Gastrointest Surg
. 2015
Aug
;19
(8
):1528
–36
.
[PubMed]
1091-255X11.
Jajja
MR
, Maxwell
D
, Hashmi
SS
, Meltzer
RS
, Lin
E
, Sweeney
JF
, et al. Standardization of operative technique in minimally invasive right hepatectomy: improving cost-value relationship through value stream mapping in hepatobiliary surgery
. HPB (Oxford)
. 2019
May
;21
(5
):566
–73
.
[PubMed]
1365-182X12.
Reiley
CE
, Lin
HC
, Yuh
DD
, Hager
GD
. Review of methods for objective surgical skill evaluation
. Surg Endosc
. 2011
Feb
;25
(2
):356
–66
.
[PubMed]
0930-279413.
Mazomenos
EB
, Chang
PL
, Rolls
A
, Hawkes
DJ
, Bicknell
CD
, Vander Poorten
E
, et al. A survey on the current status and future challenges towards objective skills assessment in endovascular surgery
. J Med Robot Res
. 2016
;1
(3
):1640010
. 2424-905X14.
Azevedo-Coste
C
, Pissard-Gibollet
R
, Toupet
G
, Fleury
É
, Lucet
JC
, Birgand
G
. Tracking Clinical Staff Behaviors in an Operating Room
. Sensors (Basel)
. 2019
May
;19
(10
):2287
.
[PubMed]
1424-822015.
Khan
RS
, Tien
G
, Atkins
MS
, Zheng
B
, Panton
ON
, Meneghetti
AT
. Analysis of eye gaze: do novice surgeons look at the same location as expert surgeons during a laparoscopic operation?
Surg Endosc
. 2012
Dec
;26
(12
):3536
–40
.
[PubMed]
0930-279416.
Twinanda
AP
, Shehata
S
, Mutter
D
, Marescaux
J
, de Mathelin
M
, Padoy
N
. EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos
. IEEE Trans Med Imaging
. 2017
Jan
;36
(1
):86
–97
.
[PubMed]
0278-006217.
Chen
W
, Feng
J
, Lu
J
, Zhou
J
. Endo3D: Online Workflow Analysis for Endoscopic Surgeries Based on 3D CNN and LSTM. OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis. Springer Lecture Notes in Computer Science. 2018
.18.
Jin
Y
, Dou
Q
, Chen
H
, Yu
L
, Qin
J
, Fu
CW
, et al. SV-RCNet: Workflow Recognition From Surgical Videos Using Recurrent Convolutional Network
. IEEE Trans Med Imaging
. 2018
May
;37
(5
):1114
–26
.
[PubMed]
0278-006219.
Funke
I
, Bodenstedt
S
, Oehme
F
, von Bechtolsheim
F
, Weitz
J
, Speidel
S
. Using 3D Convolutional Neural Networks to Learn Spatiotemporal Features for Automatic Surgical Gesture Recognition in Video.
Conference on Medical Image Computing and Computer-Assisted Intervention
. 2019
. 20.
Bodenstedt
S
, Rivoir
D
, Jenke
A
, Wagner
M
, Breucha
M
, Müller-Stich
B
, et al. Active learning using deep Bayesian networks for surgical workflow analysis
. Int J CARS
. 2019
Jun
;14
(6
):1079
–87
.
[PubMed]
1861-641021.
Liu
D
, Jiang
T
. Deep Reinforcement Learning for Surgical Gesture Segmentation and Classification.
Conference on Medical Image Computing and Computer-Assisted Intervention
. 2018
; vol 11073
, pp 247
–255
. 22.
Despinoy
F
, Bouget
D
, Forestier
G
, Penet
C
, Zemiti
N
, Poignet
P
, et al. Unsupervised Trajectory Segmentation for Surgical Gesture Recognition in Robotic Training
. IEEE Trans Biomed Eng
. 2016
Jun
;63
(6
):1280
–91
.
[PubMed]
0018-929423.
Van Amsterdam
B
, Nakawala
H
, De Momi
E
, Stoyanov
D
. Weakly Supervised Recognition of Surgical Gestures.
IEEE International Conference on Robotics and Automation
. 2019
; pp 9565
–9571
.24.
Van Amsterdam
B
, Clarkson
MJ
, Stoyanov
D
. Multi-Task Recurrent Neural Network for Surgical Gesture Recognition and Progress Prediction.
IEEE International Conference on Robotics and Automation
. 2020
.25.
Vedual
SS
, Ishii
M
, Hager
GD
. Objective assessment of surgical technical skill and competency in the operating room." Annual review of biomedical engineering vol 19, pp 301-325, 2017
.26.
Nguyen
XA
, Ljuhar
D
, Pacilli
M
, Nataraja
RM
, Chauhan
S
. Surgical skill levels: classification and analysis using deep neural network model and motion signals
. Comput Methods Programs Biomed
. 2019
Aug
;177
:1
–8
.
[PubMed]
0169-260727.
Ismail Fawaz
H
, Forestier
G
, Weber
J
, Idoumghar
L
, Muller
PA
. Accurate and interpretable evaluation of surgical skills from kinematic data using fully convolutional neural networks
. Int J CARS
. 2019
Sep
;14
(9
):1611
–7
.
[PubMed]
1861-641028.
Wang
Z
, Majewicz Fey
A
. Deep learning with convolutional neural network for objective skill evaluation in robot-assisted surgery
. Int J CARS
. 2018
Dec
;13
(12
):1959
–70
.
[PubMed]
1861-641029.
Fawaz
HI
, Forestier
G
, Weber
J
, Idoumghar
L
, Muller
PA
. Automated Performance Assessment in Transoesophageal Echocardiography with Convolutional Neural Networks.
Conference on Medical Image Computing and Computer Assisted Intervention
. 2018
.30.
Abdi
AH
, Luong
C
, Tsang
T
, Allan
G
, Nouranian
S
, Jue
J
, et al. Automatic Quality Assessment of Echocardiograms Using Convolutional Neural Networks: Feasibility on the Apical Four-Chamber View. IEEE Transactions on Medical Imaging. 20147; vol 36, no 6, pp 1221-1230.31.
Castellino
RA
. Computer aided detection (CAD): an overview
. Cancer Imaging
. 2005
Aug
;5
(1
):17
–9
.
[PubMed]
1740-502532.
Ahmad
OF
, Soares
AS
, Mazomenos
E
, Brandao
P
, Vega
R
, Seward
E
, et al. Artificial intelligence and computer-aided diagnosis in colonoscopy: current evidence and future directions
. Lancet Gastroenterol Hepatol
. 2019
Jan
;4
(1
):71
–80
.
[PubMed]
2468-125333.
Brandao
P
, Zisimopoulos
O
, Mazomenos
E
, Ciuti
G
, Bernal
J
, Visentini-Scarzanella
M
, et al. Towards a computed-aided diagnosis system in colonoscopy: automatic polyp segmentation using convolution neural networks
. J Med Robot Res
. 2018
;3
(2
):1840002
. 2424-905X34.
Hussein
M
, Puyal
J
, Brandao
P
, Toth
D
, Sehgal
V
, Everson
M
, et al. Deep neural network for the detection of early neoplasia in barret’s oesophagus
. Gastrointest Endosc
. 2020
. 0016-510735.
Janatka
M
, Sridhar
A
, Kelly
J
, Stoyanov
D
. Higher order of motion magnification for vessel localisation in surgical video.
Conference on Medical Image Computing and Computer-Assisted Intervention
. 2018
. 36.
Mascagni
P
, Fiorillo
C
, Urade
T
, Emre
T
, Yu
T
, Wakabayashi
T
, et al. Formalizing video documentation of the Critical View of Safety in laparoscopic cholecystectomy: a step towards artificial intelligence assistance to improve surgical safety
. Surg Endosc
. 2019
Oct
;•••
:1
–6
.
[PubMed]
0930-279437.
He
Q
, Bano
S
, Ahmad
OF
, Yang
B
, Chen
X
, Valdastri
P
, et al. Deep learning-based anatomical site classification for upper gastrointestinal endoscopy
. Int J CARS
. 2020
Jul
;15
(7
):1085
–94
.
[PubMed]
1861-641038.
D’Ettorre
C
, Dwyer
G
, Du
X
, Chadebecq
F
, Vasconcelos
F
, De Momi
E
, et al. Automated pick-up of suturing needles for robotic surgical assistance.
IEEE International Conference on Robotics and Automation
. 2018
; pp 1370
-1377
. 39.
Du
X
, Kurmann
T
, Chang
PL
, Allan
M
, Ourselin
S
, Sznitman
R
, et al. Articulated multi-instrument 2-D pose estimation using fully convolutional networks
. IEEE Trans Med Imaging
. 2018
May
;37
(5
):1276
–87
.
[PubMed]
0278-006240.
Colleoni
E
, Edwards
P
, Stoyanov
D
. Synthetic and Real Inputs for Tool Segmentation in Robotic Surgery.
Conference on Medical Image Computing and Computer Assisted Intervention
. 2020
. 41.
Bouget
D
, Allan
M
, Stoyanov
D
, Jannin
P
. Vision-based and marker-less surgical tool detection and tracking: a review of the literature
. Med Image Anal
. 2017
Jan
;35
:633
–54
.
[PubMed]
1361-841542.
Maier-Hein
L
, Mountney
P
, Bartoli
A
, Elhawary
H
, Elson
D
, Groch
A
, et al. Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery
. Med Image Anal
. 2013
Dec
;17
(8
):974
–96
.
[PubMed]
1361-841543.
Liu
X
, Zheng
Y
, Killeen
B
, Ishii
M
, Hager
GD
, Taylor
RH
, et al. Extremely dense point correspondences using a learned feature descriptor.
2020
; https://arxiv.org/abs/2003.0061944.
Bano
S
, Vasconcelos
F
, Tella Amo
M
, Dwyer
G
, Gruijthuijsen
C
, Deprest
J
, et al. Deep Sequential Mosaicking of Fetoscopic Videos.
Conference on Medical Image Computing and Computer Assisted Intervention
. 2019
.45.
Rau
A
, Edwards
PJ
, Ahmad
OF
, Riordan
P
, Janatka
M
, Lovat
LB
, et al. Implicit domain adaptation with conditional generative adversarial networks for depth prediction in endoscopy
. Int J CARS
. 2019
Jul
;14
(7
):1167
–76
.
[PubMed]
1861-641046.
Ma
R
, Wang
R
, Pizer
S
, Rosenman
J
, McGill
SK
, Frahm
JM
. Real-time 3d reconstruction of colonoscopic surfaces for determining missing regions.
Conference on Medical Image Computing and Computer-Assisted Intervention
. 2019
. 47.
Liu
X
, Stiber
M
, Huang
J
, Ishii
M
, Hager
GD
, Taylor
RH
, et al. Reconstructing Sinus Anatomy from Endoscopic Video—Towards a Radiation-free Approach for Quantitative Longitudinal Assessment.
2020
; https://arxiv.org/abs/2003.0850248.
Peters
TM
, Linte
CA
, Yaniv
Z
, Williams
J
. Mixed and augmented reality in medicine
. Boca Raton
: CRC Press
; 2018
. 49.
Luo
H
, Hu
Q
, Jia
F
. Details preserved unsupervised depth estimation by fusing traditional stereo knowledge from laparoscopic images
. Healthc Technol Lett
. 2019
Nov
;6
(6
):154
–8
.
[PubMed]
2053-371350.
Colleoni
E
, Moccia
S
, Du
X
, De Momi
E
, Stoyanov
D
. Deep learning based robotic tool detection and articulation estimation with spatio-temporal layers
. IEEE Robot Autom Lett
. 2019
;4
(3
):2714
–21
. 2377-376651.
Park
BJ
, Hunt
SJ
, Martin
C
3rd, Nadolski
GJ
, Wood
BJ
, Gade
TP
. Augmented and Mixed Reality: Technologies for Enhancing the Future of IR
. J Vasc Interv Radiol
. 2020
Jul
;31
(7
):1074
–82
.
[PubMed]
1051-044352.
Chen
F
, Wu
D
, Liao
H.
Registration of CT and ultrasound images of the spine with neural network and orientation code mutual information.
Medical imaging and augmented reality. 2016
; vol 9805, pp 292–301. 53.
Brunet
JN
, Mendizabal
A
, Petit
A
, Golse
N
, Vibert
E
, Cotin
S
. Physics-based deep neural network for augmented reality during liver surgery.
Conference on Medical Image Computing and Computer Assisted Intervention
. 2019
. 54.
Vercauteren
T
, Unberath
M
, Padoy
N
, Navab
N
. CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer Assisted Interventions
. Proc IEEE Inst Electr Electron Eng
. 2020
Jan
;108
(1
):198
–214
.
[PubMed]
0018-921955.
Srivastav
V
, Issenhuth
T
, Abdolrahim
K
, de Mathelin
M
, Gangi
A
, Padoy
N
. MVOR: A multi-view RGB-D operating room dataset for 2D and 3D human pose estimation.
Conference on Medical Image Computing and Computer Assisted Intervention
. 2018
.56.
Issenhuth
T
, Srivastav
V
, Gangi
A
, Padoy
N
. Face detection in the operating room: comparison of state-of-the-art methods and a self-supervised approach
. Int J CARS
. 2019
Jun
;14
(6
):1049
–58
.
[PubMed]
1861-641057.
Yengera
G
, Mutter
D
, Marescaux
J
, Padoy
N
. Less is more: surgical phase recognition with less annotations through self-supervised pre-training of CNN–LSTM networks.
2018
; https://arxiv.org/abs/1805.0856958.
Clancy
NT
, Jones
G
, Maier-Hein
L
, Elson
DS
, Stoyanov
D
. Surgical spectral imaging.
Medical Image Analysis. 202© 2020 S. Karger AG, Basel
2020
Copyright / Drug Dosage / Disclaimer
Copyright: All rights reserved. No part of this publication may be translated into other languages, reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, microcopying, or by any information storage and retrieval system, without permission in writing from the publisher.
Drug Dosage: The authors and the publisher have exerted every effort to ensure that drug selection and dosage set forth in this text are in accord with current recommendations and practice at the time of publication. However, in view of ongoing research, changes in government regulations, and the constant flow of information relating to drug therapy and drug reactions, the reader is urged to check the package insert for each drug for any changes in indications and dosage and for added warnings and precautions. This is particularly important when the recommended agent is a new and/or infrequently employed drug.
Disclaimer: The statements, opinions and data contained in this publication are solely those of the individual authors and contributors and not of the publishers and the editor(s). The appearance of advertisements or/and product references in the publication is not a warranty, endorsement, or approval of the products or services advertised or of their effectiveness, quality or safety. The publisher and the editor(s) disclaim responsibility for any injury to persons or property resulting from any ideas, methods, instructions or products referred to in the content or advertisements.
You do not currently have access to this content.
