Use of an Artificial Intelligence-Generated Vascular Severity Score Improved Plus Disease Diagnosis in Retinopathy of Prematurity

Objective To evaluate whether providing clinicians with an artificial intelligence-based vascular severity score (AI-VSS) improves consistency in diagnosis of plus disease in retinopathy of prematurity (ROP). Design This is a multi-reader diagnostic accuracy imaging study. Participants Eleven ROP experts (4 pediatric ophthalmologists, 7 retina specialists), 9 of which had been in practice for 10 or more years. Methods Retcam (Natus Medical Incorporated) fundus images were obtained from premature infants during routine ROP screening as part of the Imaging and Informatics in ROP study between January 2012 and July 2020. From all available exams, a subset of 150 eye exams from 110 infants were selected for grading. An AI-VSS was assigned to each set of images using the i-ROP DL system. The clinicians were asked to diagnose plus disease for each exam and assign an estimated VSS (range 1–9) at baseline, and then again one month later with AI-VSS assistance. A reference standard diagnosis (RSD) was assigned to each eye exam from the i-ROP study based on 3 masked expert labels and the ophthalmoscopic diagnosis. Main Outcome Measure Mean linearly weighted kappa for plus disease diagnosis compared to the RSD. Area under the receiver operating characteristic and precision-recall curves (AUROC, AUPR) for 1–9 labels compared to RSD for plus disease. Results Expert agreement improved significantly from substantial (κ: 0.69 [0.59, 0.75]) to near perfect (κ: 0.81 [0.71, 0.86]) when AI-VSS was integrated. Additionally, there was a significant improvement in plus disease discrimination as measured by mean [95% confidence interval] AUROC (0.94 [0.92, 0.96] to 0.98 [0.96, 0.99], difference: 0.04 [0.01, 0.06]) and AUPR (0.86 [0.81, 0.90] to 0.95 [0.91, 0.97], difference: 0.09 [0.03, 0.14]). Conclusions Providing ROP clinicians with an AI-based measurement of vascular severity in ROP was associated with both improved plus disease diagnosis and improved continuous severity labeling, as compared to a reference standard diagnosis for plus disease. If implemented in practice, AI-VSS could reduce inter-observer variability and standardize treatment for infants with ROP.