Endoscopic Artificial Intelligence for Image Analysis in Gastrointestinal Neoplasms

Recently, artificial intelligence (AI) has made remarkable progress in image recognition using deep learning systems, particularly convolutional neural networks, and has been applied to various medical fields [1]. Similarly, the implementation of AI has spread to the field of gastroenterology, including gastrointestinal (GI) endoscopy. In GI endoscopy, two main roles, computer-aided detection (CADe) and diagnosis (CADx), are essential. CADe helps endoscopists identify suspected lesions, while CADx aids in characterizing these lesions, particularly in distinguishing between neoplastic and nonneoplastic types or assessing the invasion depth of GI cancers.

In colonoscopy, multiple AI systems for colorectal polyps have been implemented clinically, with several randomized controlled and prospective trials [2, 3]. Their real-world performance will soon be evident. However, in esophagogastroduodenoscopy (EGD), although CADe systems for esophageal or gastric cancer (GC) are used in clinical practice, there are fewer clinical trials than for colonoscopy. Thus, AI in EGD is still in the development stage [3].

To clarify the usefulness of AI in GI neoplasms, we searched PubMed for original articles on AI and GI neoplasms using terms: “artificial intelligence, endoscopy, esophagogastroduodenoscopy, colonoscopy, deep learning, esophageal cancer, GC, colorectal polyps.” Further articles were found via manual searches and checking references of included articles. Potentially eligible articles were screened by screening the titles and abstracts, and we downloaded those full texts. In colorectal polyps, we focused on randomized controlled trials (RCTs) of CADe systems and prospective studies of CADx systems because of many retrospective reports. Reviews, commentaries, non-English publications, articles without detailed results, and articles using AI, except for CADe and CADx systems for GI neoplasms, were excluded. These search strategies are shown in Figure 1. In this literature review, we outline the latest research focusing on CADe and CADx systems for GI neoplasms, including esophageal squamous cell carcinoma (ESCC), esophageal adenocarcinoma (EAC), GC, and colorectal polyps, and discuss the prospects of AI in gastroenterology.

Esophageal cancer is the sixth most common cause of cancer-related mortality and the eighth most common cancer worldwide [4]. The two main types of esophageal cancer are ESCC, accounting for 85% of all esophageal cancers, and EAC [5]. ESCC is the predominant type of esophageal cancer in Asia, Africa, and South America [6]. Previous reports showed that the 5-year survival rate for advanced ESCC ranges from 10 to 30%. Thus, the prognosis of advanced ESCC is poor [7]. In contrast, early-stage detection and endoscopic curative resection can improve ESCC patients’ prognosis and quality of life [8‒10]. Detecting early-stage ESCC is challenging due to minor superficial irregularities, especially under white light imaging (WLI). Image-enhanced endoscopy techniques like narrow band imaging (NBI) and blue laser imaging (BLI) aid in detecting superficial ESCC [11]. However, less experienced endoscopists may still find it challenging to detect superficial ESCC compared to their experienced endoscopists [12]. AI systems could assist in detecting and diagnosing superficial ESCC, increasing the chances for endoscopic treatment for ESCC patients.

The effectiveness of CADe systems in ESCC is well documented across various studies (Table 1) [13‒24], encompassing 10 retrospective, one prospective, and one RCT. These retrospective studies showed that CADe systems for early ESCC had sensitivity ranging from 55 to 100% and specificity from 30 to 99.4% [14‒23]. Additionally, a recent systematic review and meta-analysis found that the sensitivity and specificity were 91.2% (range, 84.3–95.2%) and 80% (range, 64.3–89.9%), respectively, in the use of CADe system for ESCC [25].

Few prospective studies and RCTs have assessed CADe systems for ESCC. Yuan et al. [13] conducted a prospective evaluation in 2023, using retrospectively collected training and test images from patients who underwent endoscopic resection for superficial ESCC and precancerous lesions. Experienced endoscopists employed the CADe system in a clinical setting, observing patients’ esophagus with or without these conditions. They reported sensitivity, specificity, and accuracy rates of 98.6%, 84.3%, and 91.4%, respectively. Moreover, in 2024, Yuan et al. [24] reported the first multicenter, tandem, double-blind RCT on the CADe system for ESCC. Patients who underwent EGD for screening were randomly assigned to either the AI-first or routine-first groups, and the same endoscopist performed the EGD on the same day. The eligible patients who were sedated before the tandem EGD, pathologists, and statistical analysts were blinded to this grouping, whereas the endoscopists knew the grouping before the tandem EGD. As a result, they found that per-lesion miss rates were 1.7% in the AI-first group versus 6.7% in the routine-first group (risk ratio, 0.25; 95% CI: 0.06–1.08; p = 0.079), while per-patient miss rates were 1.9% in the AI-first group versus 5.1% in the routine-first group (risk ratio, 0.37; 95% CI: 0.08–1.71; p = 0.40). In contrast, the detection rate in the first examination was 1.8% in the AI-first group versus 1.3% in the routine-first group (risk ratio, 1.38; 95% CI: 1.03–1.86; p = 0 · 03). Although there were no significant differences in the per-lesion and per-patient miss rates for superficial ESCC and precancerous lesions, the CADe system significantly increased the detection rate [24].

Because most reports on CADe systems for ESCC were aimed at both detecting superficial ESCC and differentiating neoplastic or nonneoplastic lesions, we categorized these reports as CADe systems for ESCC. Consequently, only three studies evaluated the performance of the CADx system for ESCC, including two retrospective studies and one prospective study (Table 1) [16, 26, 27].

Fukuda et al. [16] retrospectively evaluated AI systems for ESCC and divided them into CADx and CADe systems. They collected 23,746 images from 1,544 pathologically proven superficial ESCC and 4,587 images from 458 noncancerous and normal tissues using nonmagnified NBI and BLI. Sensitivities, specificity, and accuracy were 86%, 89%, and 88% for the CADx system and 74%, 76%, and 75% for the experts, respectively [16]. Tani et al.’s [26] 2023 prospective study first assessed a real-time CADx system for ESCC in a clinical setting. Endoscopists evaluated patients at high risk for ESCC, classifying lesions as cancerous or noncancerous under magnified NBI before the real-time CADx system diagnoses. The CADx system’s accuracy, sensitivity, and specificity were 80.6%, 68.2%, and 83.4%, respectively. The study found the AI’s diagnostic accuracy was 5.1% lower than endoscopists’, failing to prove CADx’s noninferiority for ESCC diagnosis.

EAC constitutes 14% of all esophageal cancers [5] and is the most common type of esophageal cancer in Western countries, unlike ESCC [6]. Its incidence is rising in these regions, with most EAC cases diagnosed at an advanced stage [28]. Advanced EAC patients face a grim prognosis, with a 5-year survival rate under 20%, necessitating invasive treatments like extended surgical resection [29]. Conversely, superficial EAC cases treated with endoscopic resection show improved prognoses [30].

Barrett’s esophagus (BE), defined as the replacement of squamous epithelium with metaplastic epithelium due to gastroesophageal reflux disease, is a potential precancerous lesion for EAC [31]. In general, the prevalence of BE is low, and the risk of progression from nonneoplastic BE to EAC remains unclear [32]. These BE characteristics make the detection and differentiation between nonneoplastic and neoplastic BE challenging, even for experienced endoscopists. Additionally, random biopsies are recommended for patients with long-segment BE because of the high risk of EAC [32, 33]. If we can optically detect and diagnose early EAC or neoplastic BE using AI systems, we can increase the biopsy yield, and patients with EAC may have a better prognosis.

Many studies on CADe systems for detecting early EAC and neoplastic BE have been reported, mainly in Western countries (Table 2) [34‒46]. In a previous systematic review and meta-analysis of the CADe for EAC, the pooled sensitivity and specificity were 93.1% (range 86.8–96.4%) and 86.9% (range 81.7–90.7%), respectively [25].

De Groof et al. [34] first reported a prospective CADe study for EAC in 2019. In 2020, they developed a CADe system for neoplastic BE detection, achieving sensitivities, specificities, and accuracies of 90%, 89%, and 88%, surpassing general endoscopists of 72%, 74%, and 73%, respectively [35]. The system also detected neoplastic BE with 91% accuracy in real-time endoscopic procedures in a subsequent pilot study [36]. In 2023, Abedelrahim et al. [40] conducted a multicenter prospective study with 32 neoplastic and 43 nonneoplastic BE videos, showing the CADe system’s superior sensitivity over nonexperts (93.8 vs. 63.5%; p < 0.001).

Recently, the incidence of EAC in Asian countries has increased, especially in Japan [51, 52]. Iwagami et al. [45] developed a CADe system based on Japanese EAC cases, which, in a retrospective study, demonstrated a sensitivity, specificity, and accuracy of 94%, 42%, and 66%, respectively. Further studies are needed not only in Western countries but also in other countries with lower incidences of EAC.

Similar to CADx systems for ESCC, few studies have reported on CADx systems for EAC [47‒50]. Ebigbo et al. [49] (2020) developed an AI system using WLI and NBI images from a real-time camera to differentiate nonneoplastic BE from early EAC, achieving 83.7% sensitivity, 100.0% specificity, and 89.9% accuracy. Hussein et al. [47] (2023) created a CADx system for diagnosing BE dysplasia using magnifying WLI and chromoendoscopy, collecting prospective videos from patients with dysplastic and nondysplastic BE across institutions. They conducted validation tests on high-quality still images (test 1), all video frames (test 2), and selected video sequences (test 3), attaining 90% accuracy and over 90% sensitivity in all tests. Test 3 showed a diagnostic speed of 0.0135 s per frame, suggesting the system’s potential for real-time diagnosis [47].

GC is the fourth most common cause of cancer-related deaths and the fifth most common cancer worldwide [4]. East Asian countries, including Japan, account for most GCs and are affected by Helicobacter pylori infection [53]. The 5-year overall survival rate for stage I GC exceeds 95%, while stage IV GC shows a survival rate below 20% [54], highlighting the importance of early detection and treatment for better outcomes.

EGD plays a crucial role in the diagnosis and treatment of GC [55]; however, early detection and diagnosis remain challenging. GC often has a background of atrophic gastritis caused by H. pylori infection, making the detection and diagnosis of early GC difficult because only subtle morphological changes occur in early GC lesions [56]. Missed rates for GC in screening EGD are reported at 5.7%, with less experienced endoscopists showing false-negative rates up to 11.5% [57, 58]. To address this, computer-aided detection/diagnosis (CAD) systems for GC have been actively developed.

The first CADe system for GC was developed by Hirasawa et al. in 2018 [59]. The CADe system was trained on 13,584 endoscopic still images of GC and validated on 2,296 independent still images. They showed that the CADe system had a sensitivity of 92.2% for detecting GC in image analysis. Since then, many studies on CADe systems have been conducted (Table 3) [59‒76]. Retrospective reports indicate CADe systems for GC achieved sensitivities of 58.4–100% and specificities of 79.1–97.6% [59, 64‒75]. A recent meta-analysis showed CADe system sensitivity and specificity for early GC were both 90% (95% CI: 0.86–0.93 and 0.86–0.92, respectively) [77].

A few prospective studies compared the performance to detect early GC between CADe systems endoscopists. In 2019, Luo et al. [60] first reported a prospective multicenter study to evaluate the CADe system for detecting GC and ESCC, and they demonstrated a high accuracy of 92.7% for cancer detection. Wu et al. reported several studies using a CAD system for gastric neoplasms named ENDOANGEL. ENDOANGEL has various functions, including detection of early GC (CADe system), differentiation of neoplastic and nonneoplastic lesions (CADx system), and prediction of tumor invasion depth [61, 62]. In a tandem RCT, they evaluated the miss rate of gastric neoplasms between routine EGD (routine-first group) and EGD with the CADe system (AI-first group) and showed that the miss rate of neoplasms in the AI-first group was significantly lower than that in the routine-first group (6.1 vs. 27.3%; p = 0.015). They suggested that overlooked gastric neoplastic lesions could be reduced using the CADe system [76]. In addition, a new prospective multicenter RCT using ENDOANGEL-GC, which simultaneously combines blind spot monitoring and detection systems in real time, is in progress; the detection rate of gastric neoplasms might be higher than that in previous studies [94].

Several CADx systems for GC have been developed (Table 3) [61‒63, 70, 78‒93], achieving sensitivities of 36.8–99% and specificities of 58.2–100% [70, 79‒93]. Wu et al. [61] evaluated ENDOANGEL’s diagnostic performance using magnified NBI images prospectively, achieving 100% sensitivity and 82.5% specificity, comparable to expert endoscopists. They also reported a CADx system for early GC under WLI with 92.9% sensitivity and 91.7% specificity, demonstrating high accuracy in differentiating early GC even under normal WLI [61]. In Korea, Gong et al. developed an AI system for GC classification into advanced GC, early GC, dysplasia, and nonneoplastic lesions under nonmagnified WLI. Testing on 3,976 prospectively collected images from five institutions showed the CADx system achieved 81.5% accuracy [63].

In contrast, Yuan et al. [92] first reported that the CADx system with normal WLI can classify lesions into not only neoplastic/nonneoplastic but also multiple classes. The CADx system was trained to classify lesions as early GC, advanced GC, submucosal tumors, polyps, ulcers, erosions, or normal mucosa. As a result, the overall accuracy was 85.7%, which was equivalent to that of expert endoscopists (85.1%) and higher than that of nonexpert endoscopists (78.8%) [92]. Recently, Ishioka et al. [93] developed a CADx system for early GC in Japan, which can diagnose GC under normal WLI in still images. The performance of the CADx was evaluated using still images of 150 neoplastic and 165 nonneoplastic lesions and demonstrated a superior sensitivity to that of specialists (84.7 vs. 65.8%) [93]. The performance of this endoscopic AI system was also externally validated in Singapore, demonstrating higher sensitivity in detecting high-grade dysplasia compared to endoscopists (80 vs. 29.1%; p = 0.0011) [75]. An RCT of CADx for GC has not yet been conducted, and future studies are necessary to evaluate the performance of CADx systems for GC in clinical settings.

Colorectal cancer is a major global cause of cancer-related deaths, and the complete removal of tumor lesions during colonoscopy is crucial for its prevention [95, 96]. However, the detection rate of lesions during colonoscopy is highly dependent on the performance of endoscopists, particularly less experienced ones, posing a challenge to achieving a high adenoma detection rate (ADR) [97]. A decrease in ADR is associated with an increase in colorectal cancer incidence [98]. Therefore, standardizing the quality of examinations and reducing oversight of tumor lesions is essential, irrespective of the endoscopist’s experience.

Furthermore, qualitative endoscopic diagnosis of discovered lesions relies on endoscopists. While opportunities for qualitative diagnosis are increasing with the growing number of colorectal polyp cases, performing highly accurate endoscopic diagnoses, especially by nonspecialists, is considered challenging [99]. Computer assistance using AI has been developed to mitigate human-dependent errors and ensure the quality of examinations. The following section discusses the current state of research on CAD systems for colorectal polyps, with a focus on prospective studies.

Research on CADe for colorectal lesions began in the early 2000s [100, 101], but faced challenges like small sample sizes and high false positives. The adoption of deep learning in the late 2010s significantly improved speed and accuracy, revitalizing CADe interest. Misawa et al.’s [102] 2018 CADe reached 90% sensitivity. Later studies showed CADe’s sensitivity and specificity exceed 90% in real-time analysis, underscoring its utility [103, 104].

Against the backdrop of favorable outcomes in retrospective trials, several prospective RCTs have been conducted using CADe systems for colorectal polyps (Table 4) [105‒127]. Wang et al. [105] reported that the use of the CADe system significantly increased ADR in a single-center RCT involving a total of 1,058 participants (29.1 vs. 20.3%; p < 0.001). Similarly, multiple reports indicated a significant increase in ADR or polyp detection rate (PDR) with CADe using different algorithms [106‒109, 111]. Shaukat et al. [115] reported that the adenomas per colonoscopy significantly increased with the use of CADe (1.05 vs. 0.83; p = 0.002). Additionally, several studies reported a reduction in adenoma miss rate with the use of CADe [110, 112, 116, 117]. Furthermore, Wang et al. [111] reported a blind RCT using sham AI with intentionally false outputs, demonstrating that the CADe system contributed to an increase in ADR (34 vs. 28%; p = 0.030), while Xu et al. [113] reported no increase in PDR despite using CADe (38.8 vs. 36.2%; p = 0.183).

A common feature observed in these studies is that CADe increases the detection rate of relatively low-grade lesions with a diameter of less than 10 mm or sessile serrated lesions [105‒115, 128]. However, the detection rate of high-grade lesions with a diameter of 10 mm or more did not change significantly using CADe [105‒115]. The CADe system by Gong et al. [108] enhances the detection of larger lesions and also tracks insertion, observation times, and colon preparation, indicating that lesion detection is influenced by both endoscopist performance and colonoscopy quality. Ahmad et al. [119] found that expert endoscopists using CADe increased PDR (85.7 vs. 79.7%; p = 0.05) but not ADR (71.4 vs. 65.0%; p = 0.09), suggesting CADe’s effectiveness varies with endoscopist skill. An RCT by Yamaguchi et al. [127] showed CADe aids trainees’ colonoscopy proficiency, potentially reducing missed adenomas, highlighting its benefit for novices.

Multiple CADe systems have been approved for use in various countries. Although RCTs using these commercial CADe systems have been reported, the results remain controversial [119‒127]. Therefore, further large-scale studies using these systems are required. These studies are expected to clarify the specific targets and conditions under which CADe is useful.

Most polyps detected during colonoscopy are small and have a low probability of developing into advanced neoplasia or invasive cancer. Therefore, the implementation of CADx to accurately diagnose lesions could enable strategies such as “leave-in-situ” (opting not to resect diminutive polyps) or “resect-and-discard” (skipping pathological diagnosis for small lesions) approaches, which have the potential to significantly reduce costs and enhance efficiency [129].

Table 5 summarizes prospective studies conducted on CADx for colorectal polyps [130‒139]. Currently, NBI and BLI are the most widely used CADx modalities for colorectal lesions. These techniques involve image-enhanced observation methods that allow for the detailed evaluation of surface vessels, making them more suitable than WLI for the qualitative diagnosis of lesions. In 2011, Gross et al. [130] used a support vector machine algorithm to predict pathological diagnoses using magnified NBI images. In a prospective validation, they reported favorable results with a sensitivity of 95% and accuracy of 93% for differentiating tumor lesions. Additionally, Kominami et al. [131] reported real-time examinations using NBI with an accuracy of 97% for differentiating adenomas.

Regarding prospective studies on convolutional neural network-based CADx using image-enhanced endoscopy, van der Zander et al. [132] reported an accuracy of 95.0% in distinguishing tumor lesions under NBI. Furthermore, an algorithm developed by Houwen et al. [133] using BLI demonstrated a sensitivity of 89%, a specificity of 38%, and an overall accuracy of 79% in diagnosing neoplastic lesions, showing comparable diagnostic capabilities to endoscopists (p = 0.10). In prospective studies, the CAD-EYE system differentiated neoplastic from nonneoplastic lesions with accuracies reported by Dos Santos et al. (81.8%) and Li et al. (71.6%) [134, 135], both lower than endoscopists’ discrimination abilities, suggesting the algorithm needs improvement. These studies underscored the comparative performance of CADx systems and endoscopists, but collaboration might enhance diagnostic outcomes [136]. Additionally, CADx with NBI effectively distinguishes polyps, including sessile serrated lesions [140]. Min and colleagues proposed using linked color imaging with CADx referencing NBI, achieving 87.0% accuracy [137]. The GI Genius system, employing WLI, reported an 84.0% accuracy [138]. These findings underline the necessity for further large-scale studies.

As a crucial modality for the discrimination of neoplasms, the utility of magnifying endoscopy is suggested; previous studies demonstrated that NBI magnifying colonoscopy accurately differentiates the pathology of polyps with an accuracy of 86.6–97.5% [130, 131, 141]. Moreover, endocytoscopy, an ultramagnifying endoscopy technique, is noteworthy. Mori et al. [142] referenced features of nuclear areas obtained from magnified images and reported an accuracy of 89.2% in distinguishing tumor lesions. After successive improvements to the algorithm, Mori and colleagues conducted a prospective trial using endocytoscopy, reporting a sensitivity of 93.8% and specificity of 91.0% when combining NBI to discriminate lesions with 5 mm or less diameter [139]. They also suggested the utility of CADx for diagnosing lesion depth [143], indicating the need for further validation through additional prospective trials.

Recent research on AI systems for GI endoscopy is summarized. Studies typically indicate that these systems outperform nonexpert endoscopists and match or surpass expert endoscopists in effectiveness. However, the predominance of retrospective studies suggests a need for more prospective research to fully evaluate endoscopic AI systems’ clinical applications. The American Society of Gastrointestinal Endoscopy has outlined a framework for AI development, evaluation, and real-time performance metrics [144].

In contrast, several problems should be solved soon. Firstly, the cost-effectiveness of endoscopic AI systems is uncertain. Reports indicate AI’s potential cost-effectiveness for colorectal polyps in reducing colorectal cancer incidence and unnecessary polypectomies [145, 146], but limited studies exist for AI in EGD [147], necessitating further research. Secondly, ethical considerations surround endoscopic AI’s use. In Japan, guidelines by the Japan Gastroenterological Endoscopy Society recommend endoscopists using AI have ample diagnostic experience. Similarly, the American Medical Association House of Delegates advises AI should augment rather than replace human intelligence.

Endoscopists will be required to use endoscopic AI systems for detection and diagnosis, as these can reduce the risk of overlooking lesions and equalize the diagnostic ability of endoscopists. In the future, early detection and diagnosis of GI neoplasms may improve prognosis using AI systems in gastroenterology.

T.T. is the CEO of AI Medical Service Inc. T.O. and J.S. are advisory members of AI Medical Service Inc.

This research received no external funding.

R.K., K.O., and T.T. conceived and designed the study. R.K., K.O., and T.O. drafted the manuscript. J.S., S.I., and T.T. contributed to the revision and editing. All the authors have approved the final manuscript for publication.