Systematic review and meta-analysis of the readability of large language models for patient education on ophthalmological conditions

Objective Ophthalmological patient education materials (PEMs) are written at a reading level of grades 10.4–12.6, higher than the recommended 8th-grade level. This study quantitatively assesses the readability of large language model (LLM)-generated chatbot responses to patient queries, de novo PEM creation, and simplification of existing PEMs. Methods This systematic review and meta-analysis article was registered in PROSPERO (CRD420251123401). We searched Ovid MEDLINE, Embase, CINAHL, Web of Science, and Scopus (August 11, 2025). The primary outcome was readability, measured by the Flesch-Kincaid Grade Level (FKGL). Random-effects meta-analysis compared ChatGPT-generated PEM readability with existing control PEMs (i.e., clinician- or organization-authored websites, brochures, and clinical guidelines). A binomial test evaluated the proportion of studies achieving an 8th-grade reading level. Joanna Briggs Institute and Grading of Recommendations Assessment, Development, and Evaluation tools assessed risk of bias and quality. Results Of 1660 studies screened, 57 were included, with 11 contributing to the meta-analysis. Pooled FKGL scores were 9.82 (95% CI: 7.93–11.70) for ChatGPT PEMs and 9.54 (95% CI: 8.68–10.40) for control PEMs. There was no significant difference in FKGL readability between ChatGPT and control PEMs (mean difference = 0.21; 95% CI: −1.52 to 1.94; p = 0.81; I 2 = 97.08%). Seventeen studies (29.82%) included LLM(s) at or below 8th-grade reading level. Grade-level prompting significantly improved LLM-generated PEM readability. Risk of bias was low to moderate. Reasons for downgrading were inconsistency and imprecision. Conclusions In the context of high heterogeneity and low certainty, it appears that LLMs do not produce ophthalmology PEMs that are more readable than existing materials.