Abstract
Background: Application and development of the artificial intelligence technology have generated a profound impact in the field of medical imaging. It helps medical personnel to make an early and more accurate diagnosis. Recently, the deep convolution neural network is emerging as a principal machine learning method in computer vision and has received significant attention in medical imaging. Key Message: In this paper, we will review recent advances in artificial intelligence, machine learning, and deep convolution neural network, focusing on their applications in medical image processing. To illustrate with a concrete example, we discuss in detail the architecture of a convolution neural network through visualization to help understand its internal working mechanism. Summary: This review discusses several open questions, current trends, and critical challenges faced by medical image processing and artificial intelligence technology.
References
1.
Krizhevsky
A
, Sutskever
I
, Hinton
GE
. ImageNet classification with deep convolution neural networks
. In: Pereira
F
, Burges
CJC
, Bottou
L
, Weinberger
KQ
, editors. Advances in neural information processing systems 25
; 2012
. p. 1097
–105
.2.
Simonyan
K
, Zisserman
A
. Very deep convolution networks for large scale image recognition
. In: International conference on learning representations
; 2015
.3.
He
K
, Zhang
X
, Ren
S
, Sun
J
. Deep residual learning for image recognition
. The IEEE conference on computer vision and pattern recognition (CVPR)
; 2016
.4.
McCarthy
J
, Minsky
ML
, Rochester
N
, Shannon
CE
. A proposal for the Dartmouth summer research project on artificial intelligence, August 31, 1955
. AI Mag
. 2006
;27
(4
):12
.5.
Deng
J
, Dong
W
, Socher
R
, Li
LJ
, Li
K
, Fei-Fei
L
. ImageNet: a large scale hierarchical image database
. 2009 IEEE conference on computer vision and pattern recognition
. IEEE
; 2009
. p. 248
–55
.6.
Hinton
GE
, Osindero
S
, Teh
YW
. A fast learning algorithm for deep belief nets
. Neural Comput
. 2006
;18
(7
):1527
–54
. .7.
Hinton
G
. Deep belief networks
. Scholarpedia
. 2009
;4
(5
):5947
. .8.
LeCun
Y
, Bengio
Y
, Hinton
G
. Deep learning
. Nature
. 2015
;521
(7553
):436
–44
. .9.
Schuster
M
, Paliwal
KK
. Bidirectional recurrent neural networks
. IEEE Trans Signal Process
. 1997
;45
(11
):2673
–81
. .10.
Hou
L
, Samaras
D
, Kurc
TM
, Gao
Y
, Davis
JE
, Saltz
JH
. Patch-based convolution neural network for whole slide tissue image classification
. In: Proceedings of the IEEE conference on computer vision and pattern recognition
; 2016
. p. 2424
–33
.11.
Rajpurkar
P
, Irvin
J
, Zhu
K
, Yang
B
, Mehta
H
, Duan
T
, CheXNet: radiologist-level pneumonia detection on chest X-rays with deep learning
. arXiv preprint arXiv:171105225. 2017
.12.
Gulshan
V
, Peng
L
, Coram
M
, Stumpe
MC
, Wu
D
, Narayanaswamy
A
, Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs
. JAMA
. 2016
;316
(22
):2402
–10
. .13.
Girshick
R
, Donahue
J
, Darrell
T
, Malik
J
. Rich feature hierarchies for accurate object detection and semantic segmentation
. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
; 2014
. p. 580
–87
.14.
He
K
, Gkioxari
G
, Dollár
P
, Girshick
R
. Mask r-cnn
. In: Proceedings of the IEEE International Conference on Computer Vision
; 2017
. p. 2961
–69
.15.
Collobert
R
, Weston
J
, Bottou
L
, Karlen
M
, Kavukcuoglu
K
, Kuksa
P
. Natural language processing (almost) from scratch
. J Mach Learn Res
. 2011
;12
(Aug
):2493
–537
.16.
Manning
CD
, Surdeanu
M
, Bauer
J
, Finkel
JR
, Bethard
S
, McClosky
D
. The Stanford CoreNLP natural language processing toolkit
. In: Proceed-ings of 52nd annual meeting of the association for computational linguistics: system demonstrations
; 2014
. p. 55
–60
.17.
Fukushima
K
. Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position
. Biol Cybern
. 1980
;36
(4
):193
–202
.18.
Fukushima
K
. Analysis of the process of visual pattern recognition by the neocognitron
. Neural Networks
. 1989
;2
(6
):413
–20
. .19.
LeCun
Y
, Boser
B
, Denker
JS
, Henderson
D
, Howard
RE
, Hubbard
W
, Backpropagation applied to handwritten zip code recognition
. Nat Commun
. 1989
;1
(4
):541
–51
. .20.
LeCun
Y
, Bottou
L
, Bengio
Y
, Haffner
P
. Gradient-based learning applied to document recognition
. Proc IEEE
. 1998
;86
(11
):2278
–324
. .21.
Girshick
R
. Fast r-cnn
. In: Proceedings of the IEEE International Conference on Computer Vision
; 2015
. p. 1440
–48
.22.
Redmon
J
, Divvala
S
, Girshick
R
, Farhadi
A
. You only look once: unified, real-time object detection
. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
; 2016
. p. 779
–88
.23.
Liu
W
, Anguelov
D
, Erhan
D
, Szegedy
C
, Reed
S
, Fu
CY
, Ssd: single shot multibox detector
. European Conference on Computer Vision
. Springer
; 2016
. p. 21
–37
.24.
Chen
LC
, Papandreou
G
, Kokkinos
I
, Murphy
K
, Yuille
AL
. DeepLab: semantic image segmentation with deep convolution nets, atrous convolution, and fully connected CRFS
. IEEE Trans Pattern Anal Mach Intel
. 2017
;40
(4
):834
–48
.25.
Badrinarayanan
V
, Kendall
A
, Cipolla
R
. SegNet: a deep convolution encoder decoder architecture for image segmentation
. IEEE Trans Pattern Anal Mach Intel
. 2017
;39
(12
):2481
–95
.26.
Cordts
M
, Omran
M
, Ramos
S
, Rehfeld
T
, Enzweiler
M
, Benenson
R
, The cityscapes dataset for semantic urban scene understanding
. In: Proceedings of the IEEE conference on computer vision and pattern recognition
; 2016
, p. 3213
–23
.27.
Lin
G
, Milan
A
, Shen
C
, Reid
I
. RefineNet: multi-path refinement networks for high-resolution semantic segmentation
. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
; 2017
. p. 1925
–34
.28.
Simonyan
K
, Vedaldi
A
, Zisserman
A
. Deep inside convolution networks: visualising image classification models and saliency maps
. arXiv preprint arXiv:13126034. 2013.29.
Xie
Y
, Kong
X
, Xing
F
, Liu
F
, Su
H
, Yang
L
. Deep voting: a robust approach toward nucleus localization in microscopy images
. International conference on medical image computing and computer-ssisted Intervention
. Springer
; 2015
. p. 374
–82
.30.
Sanyal
P
, Mukherjee
T
, Barui
S
, Das
A
, Gangopadhyay
P
. Artificial intelligence in cytopathology: a neural network to identify papillary carcinoma on thyroid fine-needle aspiration cytology smears
. J Pathol Inform
. 2018
;9
:43
. .31.
Sanghvi
AB
, Allen
EZ
, Callenberg
KM
, Pantanowitz
L
. Performance of an artificial intelligence algorithm for reporting urine cytopathology
. Cancer Cytopathol
. 2019 Oct
;127
(10
):658
–66
. .32.
Sadanandan
SK
, Ranefall
P
, Le Guyader
S
, Wählby
C
. Automated training of deep convolutional neural networks for cell segmentation
. Sci Rep
. 2017
;7
(1
):7860
–7
. .33.
Liang
Z
, Powell
A
, Ersoy
I
, Poostchi
M
, Silamut
K
, Palaniappan
K
, CNN-based image analysis for malaria diagnosis
. In: 2016 IEEE international conference on bioinformatics and biomedicine (BIBM)
. IEEE
; 2016 Dec 15
. p. 493
–6
.34.
Li
X
, Li
W
, Xu
X
, Hu
W
. Cell classification using convolutional neural networks in medical hyperspectral imagery
. In: 2017 2nd international conference on image, vision and computing (ICIVC)
. IEEE
; 2017 Jun 2
. p. 501
–4
.35.
Ciresan
DC
, Giusti
A
, Gambardella
LM
, Schmidhuber
J
. Mitosis detection in breast cancer histology images with deep neural networks
. International conference on medical image computing and computer-assisted intervention
. Springer
; 2013
. p. 411
–8
.36.
Araújo
T
, Aresta
G
, Castro
E
, Rouco
J
, Aguiar
P
, Eloy
C
, Classification of breast cancer histology images using convolutional neural networks
. PLoS One
. 2017
;12
(6
):e0177544
. .37.
Bardou
D
, Zhang
K
, Ahmad
SM
. Classification of breast cancer based on histology images using convolutional neural networks
. IEEE Access
. 2018
;6
:24680
–93
. .38.
Li
Y
, Ping
W
. Cancer metastasis detection with neural conditional random field
. arXiv preprint arXiv: 180607064. 2018
.39.
Szegedy
C
, Vanhoucke
V
, Ioffe
S
, Shlens
J
, Wojna
Z
. Rethinking the inception architecture for computer vision
. In: Proceedings of the IEEE conference on computer vision and pattern recognition
; 2016
. p. 2818
–26
.40.
Zeng
X
, Chen
H
, Luo
Y
, Ye
W
. Automated diabetic retinopathy detection based on binocular Siamese-like convolutional neural network
. IEEE Access
. 2019
;7
:30744
–53
. .41.
Wang
X
, Peng
Y
, Lu
L
, Lu
Z
, Bagheri
M
, Summers
RM
. ChestX-ray8: hospital scale chest X-ray database and benchmarks on weakly supervised classification and localization of common thorax diseases
. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
; 2017
. p. 2097
–106
.42.
Zhou
B
, Khosla
A
, Lapedriza
A
, Oliva
A
, Torralba
A
. Learning deep features for discriminative localization
. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
; 2016
. p. 2921
–29
.43.
Li
L
, Qin
L
, Xu
Z
, Yin
Y
, Wang
X
, Kong
B
, Artificial intelligence distinguishes COVID-19 from community acquired pneumonia on chest CT
. Radiology
. 2020
:200905
.44.
Wang
L
, Wong
A
. COVID-Net: a tailored deep convolution neural network design for detection of COVID-19 cases from chest radiography images
. arXiv preprint arXiv:200309871. 2020
.45.
Apostolopoulos
ID
, Mpesiana
TA
. Covid-19: automatic detection from X-ray images utilizing transfer learning with convolution neural networks
. Phys Eng Sci Med
. 2020
;43
(2
):635
–40
.46.
Ciresan
D
, Giusti
A
, Gambardella
LM
, Schmidhuber
J
. Deep neural networks segment neuronal membranes in electron microscopy images
. Adv Neural Inf Process Syst
. 2012
;843
–51
.47.
Ronneberger
O
, Fischer
P
, Brox
T
. U-Net: convolution networks for biomedical image segmentation
. In: International Conference on Medical Image Computing and Computer-Assisted Intervention
. Springer
; 2015
. p. 234
–41
.48.
Milletari
F
, Navab
N
, Ahmadi
SA
. V-Net: fully convolution neural networks for volumetric medical image segmentation
. In: 2016 Fourth International Conference on 3D Vision (3DV)
. IEEE
; 2016
. p. 565
–71
.49.
Kamnitsas
K
, Ledig
C
, Newcombe
VF
, Simpson
JP
, Kane
AD
, Menon
DK
, Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation
. Med Image Anal
. 2017
;36
:61
–78
. .50.
Zoph
B
, Le
QV
. Neural architecture search with reinforcement learning
. arXiv preprint arXiv:161101578. 2016
.51.
Liu
C
, Chen
LC
, Schroff
F
, Adam
H
, Hua
W
, Yuille
AL
, Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation
. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
; 2019
. p. 82
–92
.52.
Hecht-Nielsen
R
. Theory of the backpropagation neural network
. Neural networks for perception
. Elsevier
; 1992
. p. 65
–93
.53.
Rumelhart
DE
, Hinton
GE
, Williams
RJ
. Learning internal representations by error propagation
. California Univ San Diego La Jolla Inst for Cognitive Science
; 1985
.54.
Rumelhart
DE
, Durbin
R
, Golden
R
, Chauvin
Y
. Backpropagation: the basic theory
. Backpropagation: theory, architectures and applications
. 1995
. p. 1
–34
.55.
Lowe
DG
. Distinctive image features from scale-invariant keypoints
. Int J Comput Vis
. 2004 Nov
;60
(2
):91
–110
. .56.
Lazebnik
S
, Schmid
C
, Ponce
J
. Beyond bags of features: spatial pyramid matching for recognizing natural scene categories
. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)
; 2006
. Vol. 2
; p. 2169
–78
.57.
Le
H
, Borji
A
. What are the receptive, effective receptive, and projective fields of neurons in convolution neural networks?
arXiv preprint arXiv:170507049. 2017
.58.
Ghrare
SE
, Mohd. Ali
MA
, Jumari
K
, Ismail
M
. An efficient low complexity lossless coding algorithm for medical images
. Am J App Sci
. 2009
;6
(8
):1502
–8
. .59.
Kil
SK
, Lee
JS
, Shen
DF
, Ryu
J
, Lee
EH
, Min
HK
, Lossless medical image compression using redundancy analysis
. 2006
.60.
Xie
C
, Wu
Y
, Maaten
Lvd
, Yuille
AL
, He
K
. Feature denoising for improving adversarial robustness
. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
; 2019
. p. 501
–9
.61.
Xie
S
, Girshick
R
, Dollár
P
, Tu
Z
, He
K
. Aggregated residual transformations for deep neural networks
. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
; 2017
. p. 1492
–1500
.62.
Szegedy
C
, Liu
W
, Jia
Y
, Sermanet
P
, Reed
S
, Anguelov
D
, Going deeper with convolutions
. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
; 2015
. p. 1
–9
.63.
Szegedy
C
, Ioffe
S
, Vanhoucke
V
, Alemi
AA
. Inception-v4, Inception-ResNet and the impact of residual connections on learning
. In: Thirty-First AAAI Conference on Artificial Intelligence
; 2017
.64.
Ma
N
, Zhang
X
, Zheng
HT
, Sun
J
. ShuffleNet v2: Practical guidelines for efficient cnn architecture design
. In: Proceedings of the European Conference on Computer Vision (ECCV)
; 2018
. p. 116
–31
.65.
Freeman
I
, Roese Koerner
L
, Kummert
A
. EffNet: an efficient structure for convolution neural networks
. In: 2018 25th IEEE International Conference on Image Processing (ICIP)
. IEEE
; 2018
. p. 6
–10
.66.
Zoph
B
, Vasudevan
V
, Shlens
J
, Le
QV
. Learning transferable architectures for scalable image recognition
. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
; 2018
. p. 8697
–710
.67.
Zhong
Z
, Yan
J
, Liu
CL
. Practical network blocks design with q-learning
. arXiv preprint arXiv:170805552. 2017
. p. 6
.68.
Liu
H
, Simonyan
K
, Yang
Y
. Darts: differentiable architecture search
. arXiv preprint arXiv:180609055. 2018
.69.
Real
E
, Moore
S
, Selle
A
, Saxena
S
, Suematsu
YL
, Tan
J
, Large scale evolution of image classifiers
. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70
. JMLR.org
; 2017
. p. 2902
–11
.70.
Xie
L
, Yuille
A
. Genetic cnn
. In: Proceedings of the IEEE International Conference on Computer Vision
; 2017
. p. 1379
–88
.71.
Haeffele
BD
, Vidal
R
. Global optimality in neural network training
. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
; 2017
. p. 7331
–39
.72.
Shwartz-Ziv
R
, Tishby
N
. Opening the black box of deep neural networks via information
. arXiv preprint arXiv:170300810. 2017
.73.
Kuo
CCJ
. Understanding convolutional neural networks with a mathematical model
. J Vis Commun Image R
. 2016
;41
:406
–13
. .74.
Baral
C
, Fuentes
O
, Kreinovich
V
. Why deep neural networks: a possible theoretical explanation
. Constraint programming and decision making: theory and applications
. Springer
; 2018
. p. 1
–5
.75.
Saxe
AM
, Bansal
Y
, Dapello
J
, Advani
M
, Kolchinsky
A
, Tracey
BD
, On the information bottleneck theory of deep learning
. J Stat Mech
. 2019
;2019
(12
):124020
. .76.
Everingham
M
, Van Gool
L
, Williams
CKI
, Winn
J
, Zisserman
A
. The Pascal visual object classes (voc) challenge
. Int J Comput Vis
. 2010
;88
(2
):303
–38
. .77.
Viola
P
, Jones
M
. Rapid object detection using a boosted cascade of simple features
. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001
. IEEE
; 2001
. Vol. 1
; p. I
.78.
Lienhart
R
, Maydt
J
. An extended set of haar-like features for rapid object detection
. In: Proceedings International Conference on Image Processing
. IEEE
; 2002
. Vol. 1
. p. I
.79.
Dalal
N
, Triggs
B
. Histograms of oriented gradients for human detection
. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)
; 2005
. Vol. 1
; p. 886
–93
.80.
Felzenszwalb
PF
, Girshick
RB
, McAllester
D
. Cascade object detection with deformable part models
. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition
. IEEE
; 2010
. p. 2241
–8
.81.
Scotti
F
. Automatic morphological analysis for acute leukemia identification in peripheral blood microscope images
. In: CIMSA. 2005 IEEE International Conference on Computational Intelligence for Measurement Systems and Applications, 2005
. IEEE
; 2005
. p. 96
–101
.82.
Rezatofighi
SH
, Soltanian-Zadeh
H
. Automatic recognition of five types of white blood cells in peripheral blood
. Comput Med Imaging Graph
. 2011
;35
(4
):333
–43
. .83.
Theera-Umpon
N
, Dhompongsa
S
. Morphological granulometric features of nucleus in automatic bone marrow white blood cell classification
. IEEE Trans Inf Technol Biomed
. 2007
;11
(3
):353
–9
. .84.
Theera-Umpon
N
. White blood cell segmentation and classification in microscopic bone marrow images
. In: International Conference on Fuzzy Systems and Knowledge Discovery
. Springer
; 2005
. p. 787
–96
.85.
Chen
M
, Xu
X
, Zhang
J
, Peng
X
, Tang
G
, Fengqi
F
, Automatic bone marrow cell classification on digital images by an artificial intelligence (AI)-based system: a pilot study
. In: Laboratory investigation
. New York, NY
: Nature Publishing Group
; 2019
. Vol. 99
.86.
Yosinski
J
, Clune
J
, Nguyen
A
, Fuchs
T
, Lipson
H
. Understanding neural networks through deep visualization
. arXiv preprint arXiv:150606579. 2015
.87.
Zeiler
MD
, Fergus
R
. Visualizing and understanding convolution networks
. European Conference on Computer Vision
. Springer
; 2014
. p. 818
–33
.88.
Shrikumar
A
, Greenside
P
, Kundaje
A
. Learning important features through propagating activation differences
. In: Proceedings of the 34th International Conference on Machine Learning
. JMLR.org
; 2017
. Vol. 70
; p. 3145
–53
.89.
Smilkov
D
, Thorat
N
, Kim
B
, Viégas
F
, Wattenberg
M
. SmoothGrad: removing noise by adding noise
. arXiv preprint arXiv:170603825. 2017
.90.
Adebayo
J
, Gilmer
J
, Muelly
M
, Goodfellow
I
, Hardt
M
, Kim
B
. Sanity checks for saliency maps
. Adv Neural Inf Process Syst
; 2018
. p. 9505
–15
.91.
Greenspan
H
, Van Ginneken
B
, Summers
RM
. Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique
. IEEE Trans Med Imaging
. 2016
;35
(5
):1153
–9
.92.
Buda
M
, Maki
A
, Mazurowski
MA
. A systematic study of the class imbalance problem in convolutional neural networks
. Neural Netw
. 2018
;106
:249
–59
. .93.
Johnson
JM
, Khoshgoftaar
TM
. Survey on deep learning with class imbalance
. J Big Data
. 2019
;6
(1
):27
. .94.
Guan
MY
, Gulshan
V
, Dai
AM
, Hinton
GE
. Who said what: modeling individual labelers improves classification
. In: Thirty-Second AAAI Conference on Artificial Intelligence
; 2018
.95.
Barlow
HB
. Unsupervised learning
. Neural Comput
. 1989
;1
(3
):295
–311
. .© 2020 S. Karger AG, Basel
2020
Copyright / Drug Dosage / Disclaimer
Copyright: All rights reserved. No part of this publication may be translated into other languages, reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, microcopying, or by any information storage and retrieval system, without permission in writing from the publisher.
Drug Dosage: The authors and the publisher have exerted every effort to ensure that drug selection and dosage set forth in this text are in accord with current recommendations and practice at the time of publication. However, in view of ongoing research, changes in government regulations, and the constant flow of information relating to drug therapy and drug reactions, the reader is urged to check the package insert for each drug for any changes in indications and dosage and for added warnings and precautions. This is particularly important when the recommended agent is a new and/or infrequently employed drug.
Disclaimer: The statements, opinions and data contained in this publication are solely those of the individual authors and contributors and not of the publishers and the editor(s). The appearance of advertisements or/and product references in the publication is not a warranty, endorsement, or approval of the products or services advertised or of their effectiveness, quality or safety. The publisher and the editor(s) disclaim responsibility for any injury to persons or property resulting from any ideas, methods, instructions or products referred to in the content or advertisements.
You do not currently have access to this content.
