Comprehensive Evaluation of ChatGPT’s Diagnostic Accuracy on Image-based Ophthalmic Case Interpretations

Objective To evaluate the diagnostic and treatment accuracy of Chat Generative Pre-trained Transformer (GPT-4.o) in ophthalmology, comparing performance when provided with full clinical context versus image-only inputs. Design Cross-sectional diagnostic accuracy study. Subjects A total of 261 ophthalmic cases from the University of Iowa's publicly available EyeRounds repository, spanning 6 subspecialties (pediatrics, retina, neuro-ophthalmology, glaucoma, oculoplastics, and cornea/cataract). Methods Each case was analyzed under 2 conditions: (1) full-context input (clinical narrative and expert-generated image descriptions) and (2) image-only input (raw clinical images without text). Chat Generative Pre-trained Transformer was prompted to identify key signs/symptoms, differential diagnoses, final diagnosis, and treatment recommendations. Outputs were compared to expert-authored case solutions and evaluated for diagnostic accuracy and component match rates. Logistic regression was used to determine which output features predicted correct diagnosis. Main Outcome Measures Primary outcome: diagnostic accuracy (match to expert diagnosis). Secondary outcomes: percentage match of signs/symptoms, differential diagnoses, and treatment recommendations. Results Diagnostic accuracy was significantly higher with full-context input (80.1%) than image-only input (54.7%) (χ 2 = 48.00, P < 0.001). Match rates for full versus image conditions were: signs/symptoms (56.5% vs. 42.6%), differential diagnoses (45.1% vs. 35.7%), treatment (64.8% vs. 49.6%), and overall match (53.8% vs. 41.5%) (all P < 0.001). In full-context cases, only treatment match significantly predicted diagnostic accuracy (odds ratio = 1.02, P < 0.001). In image-only cases, both signs/symptoms and treatment match predicted accuracy (odds ratio = 1.037 each, P < 0.001). Subspecialty analysis showed highest full-context accuracy in pediatrics (64.7%) and lowest image-only accuracy in glaucoma (21.7%) and neuro-ophthalmology (23.7%). Conclusions Chat Generative Pre-trained Transformer demonstrated high diagnostic accuracy when provided with full clinical context but significantly reduced performance with image-only input. These findings highlight the limitations of multimodal artificial intelligence (AI) in interpreting raw ophthalmic images and underscore the importance of integrating structured clinical data to optimize AI-driven diagnostic support. Financial Disclosure(s) The author has no/the authors have no proprietary or commercial interest in any materials discussed in this article.