Artificial intelligence for diabetic retinopathy screening: beyond diagnostic accuracy

Vision impairment and blindness due to diabetic retinopathy (DR) can be effectively prevented by screening programs (1); such interventions are recommended by the World Health Organization (2) and are considered very cost-effective among all the interventions to manage diabetes and its complications (3).

Traditionally, DR screening was accomplished by an individual’s retinal examination, performed annually by retinal specialists; however, with an estimated 537 million adults with diabetes in 2021 (4), such approach is not sustainable (5). Hence, in order to increase the coverage of screening programs, new alternatives have been proposed and implemented. The study “Performance of the AIDRScreening system in detecting diabetic retinopathy in the fundus photographs of Chinese patients: a prospective, multicenter, clinical study” from Dr. Yao Yang and collaborators, recently approved for publication at Annals of Translational Medicine, investigated an artificial intelligence (AI) system for the screening of DR in China (6), the country with the highest number of individuals with diabetes worldwide (4), where over 110 million patients need to be screened annually (6). In their well-conducted multicenter study, designed according to the STARD guidelines, the authors report a sensitivity of 86.72% (95% CI: 83.39–90.05%) and a specificity of 96.09% (95% CI: 94.14–97.54%) in the detection of referable DR. The accuracy was considered adequate by the Chinese regulatory authority, and the system is currently approved for use in China.

Besides presenting data that support the use of a novel tool which allows automation of screening, with an enormous potential for large-scale implementation in a cost-effective manner, while alleviating the burden of screening on the specialized workforce and hence allowing the allocation of such personnel for the treatment of detected cases, the authors also discuss some fundamental issues related to the real-world deployment of a DR screening program, such as validation with multiple camera types. One of such issues is related to the evaluation of image quality: since the validated system was not designed to evaluate quality, real-world implementation would require trained operators to do so; the authors mention that a new version of the system can assess image quality, and this ability should be evaluated in further studies. Alternatively, a semi-automated strategy with quality assessment by the photographer could be proposed.

Another real-world issue for the implementation of screening programs is related to the definition of a case. Although the authors have defined referable DR as DR equal to or worse than moderate non proliferative DR (6), diabetic macular edema (DME) is more common than proliferative DR and corresponds to the predominant cause of moderate or severe vision loss in patients with diabetes (7). Even though it is challenging to identify DME in bi-dimensional color fundus photographs, a valid alternative could be a semi-automated strategy combined with a higher sensitivity (8), with or without telemedicine evaluation by specialists (9); a two-tiered evaluation involving the more expensive optical coherence tomography (OCT) machines could be advised only for those with suspected macular edema (8). Certainly, the scarcity of ophthalmic medical resources mentioned by the authors has to be taken into account in evaluating the trade-off between sensitivity and specificity (6); since referrals incur costs for the patient and the health-care system (10), and since cost considerations are essential for the sustainability screening programmes, a formal economic evaluation would be warranted for the ideal calibrations of such operating points (9).

The study conducted by Yao Yang and colleagues involved evaluating samples in multiple centers, thus bringing more representativity to the validation of the AI system; however, all three distinct geographic locations in China (Zhongshan, Beijing, and Wenzhou) correspond to areas where the Han ethnicity is predominant. Since China is a multi-ethnic country, even though the majority of its inhabitants belong to the Han ethnicity, it would be interesting if further studies could assess the system’s performance in ethnically diverse populations.

It has been discussed that, for the real-world deployment of AI systems in health, an adequate diagnostic accuracy is necessary but not sufficient: the clinical validation cannot be considered complete unless the system is fully integrated into clinical practice. The identification of referable DR is only the first step in the patient’s journey; ultimately, the successful implementation must be measured in terms of improvement of the population’s health (7). Even if a deep-learning system generated increased screenings and better referral adherence, access to subsequent specialty care might be unavailable within the current health-care infrastructure (7). There are fundamental questions related to the integration of AI within the clinical workflow which will demand further research, including evaluation of clinical outcomes, evaluating whether all patients referred were actually treated, the ability of the local health systems to accommodate the additional number of patients identified, and whether timely care was provided (5). Formal health economic studies should evaluate if automatic systems really allow more efficient resource allocation in comparison to a semiautomated approach, for example (7,11).

Besides the availability of exams, obtaining adherence to screening and also to referral are additional challenges of screening initiatives. Along with AI, which has been reported to allow higher referral adherence from point-of-care screening recommendations (10), other tools have been proposed to achieve higher coverage and adherence rates, even in settings where eyecare is readily available, such as telemedicine (12). Telemedicine protocols, besides having been proven cost-effective for this use case (13), are all the more remarkable in China considering its dimensions, the fact that the majority of patients at high risk live in rural areas (14), and the possibility of employing mobile units and portable retinal cameras (8). Furthermore, raising awareness about DR among patients and local health-care workers is essential for increased attendance and the overall success of screening initiatives (9). Moreover, it is important that healthcare personnel are familiarized with digital health concepts, and the same applies to the end users: patients ultimately will benefit from education on how digital tools can assist decision-making in healthcare (15,16).

Even though recent advances have been occurring, DR screening is a large unmet need also among other countries where enormous contingents of individuals with diabetes dwell (4), as is the case of the United States, where less than half of such individuals adhere to eye screening guidelines (5,17), and also of other low to middle income countries, where the availability of screening programs and adherence rates are even lower (4,18-21). In this sense, the study by Yao Yang and collaborators is very relevant not only for the local setting, but on a global scale.

Therefore, notwithstanding the enormous potential to alleviate the current burden on DR screening in China and possibly elsewhere, the author’s conclusion that the AI system is safe and effective still lacks the evaluation of further aspects within the clinical workflow; future studies on real-life outcomes remain necessary. Nevertheless, the study by Yao Yang and colleagues (6) is an important contribution, addressing a very urgent topic, as the diabetes epidemic is growing, and health systems are struggling globally to tackle blindness caused by diabetes. It is clear that telemedicine programs and automatic systems will surely play a role in harnessing the digital revolution in this field, increasing access in remote or underserved areas and maximizing outcomes in a cost-effective and sustainable manner.

Acknowledgments

Funding: None.

Footnote

Provenance and Peer Review: This article was commissioned by the editorial office, Annals of Translational Medicine. The article did not undergo external peer review.

Conflicts of Interest: The author has completed the ICMJE uniform disclosure form (available at https://atm.amegroups.com/article/view/10.21037/atm-22-4756/coif). The author has no conflicts of interest to declare.

Ethical Statement: The author is accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Scanlon PH. The English National Screening Programme for diabetic retinopathy 2003-2016. Acta Diabetol 2017;54:515-25. [Crossref] [PubMed]
2. World Health Organization. Promoting Diabetic Retinopathy. (Accessed Sep 19, 2022). Available online: https://www.who.int/europe/activities/promoting-diabetic-retinopathy-screening
3. Li R, Zhang P, Barker LE, et al. Cost-effectiveness of interventions to prevent and control diabetes mellitus: a systematic review. Diabetes Care 2010;33:1872-94. [Crossref] [PubMed]
4. International Diabetes Federation. IDF Diabetes Atlas. 10th ed. Brussels, Belgium: 2021.
5. Silva PS, Aiello LP. Telemedicine and eye examinations for diabetic retinopathy: a time to maximize real-world outcomes. JAMA Ophthalmol 2015;133:525-6. [Crossref] [PubMed]
6. Yang Y, Pan J, Yuan M, et al. Performance of the AIDRScreening system in detecting diabetic retinopathy in the fundus photographs of Chinese patients: a prospective, multicenter, clinical study. Ann Transl Med 2022;10:1088. [Crossref]
7. Yuan A, Lee AY. Artificial intelligence deployment in diabetic retinopathy: the last step of the translation continuum. Lancet Digit Health 2022;4:e208-9. [Crossref] [PubMed]
8. Malerbi FK, Mendes G, Barboza N, et al. Diabetic Macular Edema Screened by Handheld Smartphone-based Retinal Camera and Artificial Intelligence. J Med Syst 2021;46:8. [Crossref] [PubMed]
9. Malerbi FK, Melo GB. Feasibility of screening for diabetic retinopathy using artificial intelligence, Brazil. Bull World Health Organ 2022;100:643-7. [Crossref] [PubMed]
10. Ruamviboonsuk P, Tiwari R, Sayres R, et al. Real-time diabetic retinopathy screening by deep learning in a multisite national screening programme: a prospective interventional cohort study. Lancet Digit Health 2022;4:e235-44. [Crossref] [PubMed]
11. Xie Y, Nguyen QD, Hamzah H, et al. Artificial intelligence for teleophthalmology-based diabetic retinopathy screening in a national programme: an economic analysis modelling study. Lancet Digit Health 2020;2:e240-9. [Crossref] [PubMed]
12. Mansberger SL, Sheppler C, Barker G, et al. Long-term Comparative Effectiveness of Telemedicine in Providing Diabetic Retinopathy Screening Examinations: A Randomized Clinical Trial. JAMA Ophthalmol 2015;133:518-25. [Crossref] [PubMed]
13. Siegel KR, Ali MK, Zhou X, et al. Cost-effectiveness of Interventions to Manage Diabetes: Has the Evidence Changed Since 2008? Diabetes Care 2020;43:1557-92. [Crossref] [PubMed]
14. Xu Y, Wang L, He J, et al. Prevalence and control of diabetes in Chinese adults. JAMA 2013;310:948-59. [Crossref] [PubMed]
15. Gray K, Slavotinek J, Dimaguila GL, et al. Artificial Intelligence Education for the Health Workforce: Expert Survey of Approaches and Needs. JMIR Med Educ 2022;8:e35223. [Crossref] [PubMed]
16. Khullar D, Casalino LP, Qian Y, et al. Perspectives of Patients About Artificial Intelligence in Health Care. JAMA Netw Open 2022;5:e2210309. [Crossref] [PubMed]
17. Liu Y, Swearingen R. Diabetic Eye Screening: Knowledge and Perspectives from Providers and Patients. Curr Diab Rep 2017;17:94. [Crossref] [PubMed]
18. Fernandes AG, Ferraz AN, Brant R, et al. Diabetic retinopathy screening and treatment through the Brazilian National Health Insurance. Sci Rep 2022;12:13941. [Crossref] [PubMed]
19. Adriono G, Wang D, Octavianus C, et al. Use of eye care services among diabetic patients in urban Indonesia. Arch Ophthalmol 2011;129:930-5. [Crossref] [PubMed]
20. Afghani T, Khan SB, Mansoor H, et al. Assessment of diabetic retinopathy and diabetes management systems in Pakistan using a WHO tool. J Pak Med Assoc 2020;70:1619-24. [Crossref] [PubMed]
21. Gupta V, Azad SV, Vashist P, et al. Diabetic retinopathy screening in the public sector in India: What is needed? Indian J Ophthalmol 2022;70:759-67. [Crossref] [PubMed]