Screening for Diabetic Retinopathy Using an Automated Diagnostic System Based on Deep Learning: Diagnostic Accuracy Assessment

Human grading telemedicine-based screening of diabetic retinopathy (DR) is the method of choice in different parts of the world [1] to early identify people with DR, so that they can be effectively treated, and, therefore, avoid vision loss. Regular screening for detection of DR is recommended for all individuals with diabetes [1]. Digital-based colour fundus images are acquired with non-mydriatic retinal cameras operated by a trained eye technician in local settings, usually at primary care offices [1], envisioning a higher rate of patient compliance [2]. Subsequent remote image evaluation by a retina specialist [3, 4] helps to deal with shortage of ophthalmologists [5]. Patients are referred to a retina specialised clinic if clinical signs of severe non-proliferative DR, proliferative DR or moderate macular oedema are shown on retinal photographs [6]. This specific condition is commonly named as referral DR. However, telescreening has some potential vulnerabilities, namely the complex and time-consuming task of interpreting eye fundus images for DR diagnosis and the low reliability between ophthalmologists in this task, varying from weak to substantial (k 0.19–0.75) [7, 8].

DR affects 34.6% of individuals with diabetes [9]. As this number is expected to increase from 382 million in 2013 to 592 million in 2035 [10], the interest in automated methods increased over the last years [11]. Current research includes the estimation of diagnostic accuracy of deep learning-based diagnostic systems [12]. The Convolutional Neural Network (CNN) model with Inception-V3 is an artificial neural network based on deep learning for automated detection of DR [13]. In a previous pilot study to determine preliminary safety and performance with individuals with diabetes referred to ophthalmology because of DR, the software achieved a sensitivity of 74% and a specificity of 95% [14]. Nevertheless, this innovative diagnostic test needs to be clinically validated in people with the target disease in similar conditions to those in which the test is intended to be used [15, 16]. For this purpose, sensitivity, specificity, predictive values and likelihood ratios should be used [15, 16]. This information is relevant for clinicians who want to apply the findings of the study to decide whether to adopt the test. We conducted a study to evaluate the diagnostic accuracy and clinical usefulness of the software in a larger population of individuals with diabetes, with and without DR, in a screening context.

A cross-sectional observational study was conducted using anonymised retinal images of individuals with diabetes from EyePACS, a publicly available dataset from primary care offices in the United States of America [17]. Images were non-stereoscopic, acquired by trained camera operators with a variety of non-mydriatic digital retinal cameras, including Canon CR-DGi and Canon CR-1, and with selective pupil dilation [17, 18]. A study sample of 350 images was randomly selected from the database containing 53,571 images.

Sample size calculation was made for a binary test outcome and the following aspects: an expected sensitivity of at least 70% [14] (given the result of a preliminary study with the neural network and also because this value is considered to be the lowest value accepted for screening tools [19]), a confidence level of 95%, a type I error of 5%, a power of 95%, a type II error of 12%, and a prevalence of DR of 20% [20]. Accordingly, the required sample size was composed of 286 images. Considering a median of image rejection rate of 18.3% [3] for fundus images because they were ungradable, the sample size was estimated as 350 images.

The 2019 CNN model with Inception-V3 in EyePACS data is an automated diagnostic system based on Deep Learning trained in EyePACS and Messidor datasets that automatically identifies DR in digital colour fundus images [13]. It provides a dichotomised classification as presence or absence of DR or referable DR. The process of image analysis is fully automatic and was performed on a computer.

The reference standard was established by taking the majority decision from 3 independent graders: the eye care specialist of the EyePACS, one general ophthalmologist and one retina specialist. They used a modified version of the International Clinical Disease Severity Scale (ICDSS) for DR, that was derived from the Early Treatment of Diabetic Retinopathy Study (ETDRS) and is used in most DR screenings [21]. Accordingly, DR was classified into 4 severity levels: without retinopathy (R0), mild non-proliferative (R1), moderate or severe pre-proliferative (R2) and proliferative (R3). Diabetic macular oedema was classified as M1. Images deemed not classifiable by at least one grader were not included in the study. Images graded as R0 were classified as negative for the presence of DR and images graded as R1, R2 or R3 or M1 as positive for DR; R0 and R1 were classified as negative for the presence of referable DR and R2, R3 or M1 as positive.

The software results were compared to those of the clinical reference standard. Sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV), with 95% CI were calculated as measures of diagnostic accuracy in accordance with the STARD 2015 guidelines for reporting diagnostic accuracy studies [22]. Sensitivity is the proportion of images identified with the disease in those with a positive test result, and specificity is the proportion of images identified without the disease in those with a negative test result. PPV is the proportion of images with a positive result that have the disease, and NPV is the proportion of images with a negative result that do not have the disease.

The likelihood ratios were also calculated because they give an estimation of how likely it is for an individual with diabetes to have DR/referable DR, thereby helping clinical decision-making [16]. The positive likelihood ratio (LR+) of a test corresponds to a ratio between the proportion of true positives (i.e., sensitivity) and the proportion of false positives (i.e., 1 – specificity) [16]. In contrast, the negative likelihood ratio (LR–) of a test corresponds to a ratio between the proportion of false negatives (i.e., 1 – sensitivity) and the proportion of true negatives (i.e., specificity) [16].

The clinical usefulness of the software test was assessed through the extent to which it helps to modify the pre-test probability of occurrence of DR/referable DR in individuals with diabetes [16]. The post-test probability was calculated using the graphical tool Fagan’s nomogram, knowing the pre-test probability (i.e., the prevalence of disease in the study population) and the likelihood ratios [16].

Of the 350 fundus images, 55 (16%) were classified as ungradable by at least one of the ophthalmologists and were excluded. All classifiable images had software automated classification as described in Figure 1.

The prevalence of DR was 15.9% (95% CI 12.0–20.7%). The software correctly identified 38 of the 47 (81%) analysable images with any DR and 38 of 47 (81%) with referable DR. Among the 248 analysable images with no DR according to the panel of ophthalmologists, there were 237 (96%) and 242 (98%) images with DR absent output for any DR and referable DR, respectively. There were 11 false positives and 9 false negatives for any DR, and 6 false positives and 9 false negatives for referable DR. The software and the reference standard classification for all 295 images are described in Table 1.

Forty-seven images had DR according to ophthalmologists grading, and sensitivity and specificity of the software in identifying DR were of 80.8 and 95.6%, respectively, with PPV and NPV of 77.6 and 96.3%, respectively. Although some of the values were higher when the clinical outcome was referable DR, no significant statistical differences were found in the observed accuracy measures between the identification of DR or referable DR. Table 2 shows the calculated sensitivity, specificity, PPV, NPV, LR+ and LR– for the software for screening of DR and referable DR with respective 95% CI.

For DR, the LR+ was of 18.2, while the LR– was 0.20. When using Fagan’s nomogram, a straight line from a patient’s pre-test probability (16%) is drawn through the LR for a positive and for a negative test result, which points to the post-test probability of DR for both cases. This led to a post-test probability for a positive test result of 74% and a post-test probability for a negative test result of 3%, as shown in Figure 2. The absolute difference between pre- and post-test probabilities of DR is 0.58 for positive and 0.13 for negative test results. Similarly, for referable DR, the LR+ was of 33, while the LR– was of 0.20. Given an estimated prevalence of 16%, if the patient tests positive, the post-test probability that he/she has truly referable DR would be 86%; if the patient tests negative, the post-test probability that he/she has truly referable DR would be 4%. The absolute difference between pre- and post-test probabilities of referable DR is 0.70 for positive and 0.12 for negative test results.

Diagnostic automation for DR using artificial intelligence will likely become essential as our rapidly ageing societies and increasing trends in diabetes prevalence continue to challenge healthcare systems with more demand for observation of fundus images. With the clinical use of the CNN model with Inception-V3 automated diagnostic system for DR, we can expect 85 fewer observations by ophthalmologists at reading centres, at a <4% probability of missing referable DR. The use of the CNN model for DR referral triage classification would safely enable to reduce ophthalmologists’ workload. If we were to refer all images automatically classified as DR, this would result in a slightly higher number of observations needed when compared with sending referable DR only (reduction of 83 vs. 85%) without an important impact on the proportion of false negatives. Therefore, we consider that the best artificial intelligence screening approach would be for ophthalmologists at the reading centre to receive images classified as referable DR-positives by the software. In this way, clinical practices could easily accommodate the expected increase in the number of required screenings worldwide. If this software is used at point-of-care at the time of image acquisition, it has the additional advantage of immediate communication of test results to the patient and more rapid observation of screen-positive patients by an ophthalmologist.

The CNN model with Inception-V3 demonstrated an 81% sensitivity for the detection of referable DR, which is considered to be safe according to the NHS’ Exeter Standard, being above the established 80% as the minimum sensitivity [23]. Only 1 in 10 patients with referable DR (4% of 38 patients) was missed by the software. Also, with a specificity of 98% for referable DR diagnosis, it exceeded the recommended value of 95% [23], showing that it correctly excludes people without the disease with 2% of false positives (6 in 248). This is an important attribute as the individuals screened positive will be advised to go to a retina specialised clinic, which represents a waste of resources. Moreover, the high specificity enables us to conclude that if a person tests positive with the software, it rules in referable DR. Values of sensitivity and specificity observed for DR were similar to the ones obtained for referable DR.

Several studies report sensitivities that vary from 87 to 97% and specificities from 59 to 98% [24]. A recent meta-analysis showed that deep learning algorithms perform well in screening for DR, having a pooled area under the receiving operating curve of 0.97 (95% CI 0.95–0.98), and a pooled sensitivity and specificity of 83% (95% CI 83–83%) and 92% (95% CI 92–92%), respectively, for detecting referable DR from fundus images, with additional reduction in misdiagnosis [25]. Studies reporting diagnostic accuracy measures of other deep learning algorithms, in the same population and using the EyePACS dataset, have reported sensitivities that ranged from 30 to 100% and from 82 to 92% for DR and referable DR, respectively, and specificities that varied from 85 and 99% and from 71 to 97% for DR and referable DR, respectively [25, 26]. When compared to the EyeArt, which showed a sensitivity and a specificity of 91% (95% CI 90.9–91.7%) for referable DR detection, the CNN model with Inception-V3 had statistically lower sensitivity, but statistically higher specificity. The IDx-DR artificial intelligence diagnostic system, clinically validated in the same population, had similar values of both sensitivity and specificity to the CNN model with Inception-V3, with a sensitivity of 87% and a specificity of 90%, therefore both higher than the minimum set by the FDA: 85 and 82.5% for sensitivity and specificity, respectively [27]. Deep learning algorithms have also been applied recently to retinal optical coherence tomography (OCT) and OCT angiography (OCTA) images for automated diagnosis of DR, because these imaging techniques provide higher sensitivity to detect early diabetic retinal changes [28]. Diagnostic accuracy studies have not been published yet, but preliminary internal validation tests show comparable diagnostic accuracy to the one using fundus images: Sandhu et al. [29] reported their algorithm to have sensitivity of 93% and specificity of 95%, and the work of Li et al. [30] achieved a sensitivity of 90% and specificity of 95%; using OCTA images, sensitivity of 98% and specificity of 87% were reported by Sandhu et al. [31], and the system of Le et al. [32] achieved 84% sensitivity and 91% specificity.

One measure that can specifically help the clinician to estimate the probability of an individual to have the disease is the predictive values [16]. A high NPV was achieved with the software – 96%, indicating the probability for a patient not to have referable DR given that the test result is negative. This value gives very high certainty to safely exclude referable DR in subjects screened negative with the CNN model with Inception-V3 and thus they do not need to be sent to a medical appointment with an ophthalmologist. A PPV of 86% reveals the probability for a positive test result to indicate the presence of referable DR, which is a good value in screening settings, where prevalence of the disease is low, making the PPV decrease [33]. To the best of our knowledge, only 2 other artificial intelligence systems described predictive values: the CNN model presented higher PPV (86 vs. 72%) than the EyeArt, and slightly lower NPV (96 vs. 98%), despite EyeArt’s values were under our CIs; the study of Kanagasingam et al. [34] reported an NPV of 100%, but a PPV of only 12% (95% CI 8–18%), which is significatively lower compared to the CNN model and demonstrating high false-positive rate, despite the high specificity of 92% (95% CI 87–96%) that is not significatively higher than the CNN model’s specificity.

When a clinician needs to interpret the test result for the individual patient in a different population, likelihood ratios, which are independent of prevalence and inherent to the test, are advantageous. A diagnostic test will be more useful to the extent that its positive likelihood ratio is of greater magnitude and that its negative likelihood ratio is smaller. In our study, for referable DR, the positive likelihood ratio was of 33 and negative likelihood ratio 0.20. Both values are statistically significantly higher than the ones reported in a systematic review: 14.11 (95% CI 9.91–20.07), and 0.10 (95% CI 0.07–0.16), for positive and negative likelihood ratios, respectively [25]. Because clinicians are more familiar with thinking in terms of probabilities, using Fagan’s nomogram to translate likelihood ratios into post-test probabilities, the result showed that after the CNN model with Inception-V3 positive result, the probability of a patient having referable DR has increased from 16 to 86%. This probability is high and indicates that referable DR is likely [35, 36] and, therefore, observation by an ophthalmologist is recommended. This value is not very high though, which is not surprising, since false positives are expected in screening. Considering the recommendation that a clinician should be at least 90% certain to make a definite diagnosis [37], and that the software has screening purposes, 86% seems to provide high certainty in relation to the presence of disease and adequateness in sending the patient for diagnosis confirmation in further observation. Likewise, with a negative CNN model with Inception-V3 result, the probability of referable DR shifts from 20 to 4%, and the disease can be excluded with reasonable certainty.

The CNN model with Inception-V3 demonstrated a good performance regarding diagnostic accuracy as an artificial intelligence screening test for referable DR, identifying and excluding disease with a good balance, enabling few both false positives and negatives. Predictive values, likelihood ratios and the post-test probability of disease showed that the software supports clinicians in determining a patient’s care pathway in screening of referable DR, helping to estimate the probability of an individual patient having or not having the disease. As such, all measures of diagnostic accuracy converge to demonstrate the adequacy and clinical usefulness of the CNN model with Inception-V3, with safety in screening decisions about individual people and advantages for healthcare systems.

A limitation of our study was that the reference standard panel was not exclusively composed of retina specialists, as one was a general ophthalmologist. Literature shows lower levels of agreement between general ophthalmologists when compared to the agreement between retina specialists [7, 8]. Additionally, the DR prevalence of the study population (16%) was lower than the value used for sample size calculation (20%); these could have affected the power of the test, with wider CIs and subsequent decrease in precision. However, the CI for prevalence includes the prevalence used for the sample size calculation. On the other hand, we assumed a sensitivity of 70% for sample size calculation, having taken into account a previous study [14] with an old version of the algorithm and conducted with patients with DR screen-positives only, which may have led to a higher precision.

Further work is needed to validate the CNN model with Inception-V3, envisioning its implementation in clinical practice, namely, through the conduction of a multicentre study with patients with diabetes followed at primary care institutions, comparing its results against a panel of retina specialists as reference standard. Assessment of its effectiveness in real-world clinical settings is also desirable. Future research should consider investigating the diagnostic accuracy of the software on retinal images acquired with different fundus cameras.

This study analysed the properties of the 2019 CNN model with Inception-V3 artificial intelligence-based automated classification of single-field digital eye fundus images acquired with table-top fundus cameras for referable DR screening. We concluded that the CNN model based on Deep Learning compared favourably with the reference standard and met the pre-established values of the Food and Drug Administration and the NHS’ Exeter Standard. Therefore, this diagnostic test accuracy study provides evidence that the software correctly identifies and rules out DR and justifies a subsequent multicentre prospective diagnostic accuracy study with an independent cohort of individuals with diabetes to enable generalization of results. Considering the clinical application of this diagnostic test for screening, it gives the clinician strong evidence to support ruling out referable DR in images classified as negative, with few false negatives. Our results suggest that the integration of the CNN model with Inception-V3 into the DR screening workflow shows promise in alleviating the great demanding and time-consuming task of ophthalmologists at reading centres.

The authors acknowledge Telmo Barbosa, MSc, Fraunhofer Portugal AICOS, for the development and management of the web annotation tool for classification of retinal images by ophthalmologists, and Tânia Borges, MD, MSc, and Gustavo Bacelar-Silva, MD, MSC, for providing image classifications. We acknowledge João Gonçalves, MSc, Fraunhofer Portugal AICOS, for providing the CNN model with Inception-V3 outputs, Ricardo Graça, MSc, Fraunhofer Portugal AICOS, for inputs on data preparation, and Ana Correia de Barros, PhD, Fraunhofer Portugal AICOS, for grammatical and stylistic revision of the manuscript. We also acknowledge Kaggle Inc. (https://www.kaggle.com/c/diabetic-retinopathy-detection/data) and EyePACS, LLC (http://www.eyepacs.com) for providing the eye fundus images dataset used in this study.

Ethical approval was not required or obtained for this study, because we used a database of retinal images collected by EyePACS, LLC and publicly available through Kaggle Inc.

S.R. and F.S. are employees of Fraunhofer Portugal AICOS, Porto, Portugal, and this institution is developing a decision-support system for diabetic retinopathy automatic classification. M.M.-S. and M.D.-M. have no financial involvement with Fraunhofer Portugal AICOS and no competing interests. No other relationships or activities could appear to have influenced the submitted work.

This work was supported by Fraunhofer Portugal AICOS (Porto, Portugal); the development of the web-platform for classification of retinal images was supported by Project MDevNet – National Network for Transfer of Knowledge of Medical Devices, in the scope of the Portuguese national programme NORTE 2020 under Portugal 2020. The sponsor or funding organization had no role in the design or conduct of this research.

S.R. conception and design, acquisition, analysis and interpretation of data, article draft, final approval of the version to be published, agreement to be accountable for all aspects of the work. M.D.-M. interpretation of data, article revision, final approval of the version to be published, agreement to be accountable for all aspects of the work. F.S. conception and design, acquisition and interpretation of data, article revision, final approval of the version to be published, agreement to be accountable for all aspects of the work. M.M.-S. conception and design, interpretation of data, article revision, final approval of the version to be published, agreement to be accountable for all aspects of the work.