An evaluation of the use of ChatGPT in production of cataract surgery patient information leaflets by doctors and patients

Background/Objectives Patient information can influence decision-making and engagement with healthcare. This study compares the quality of cataract surgery patient information leaflets (PILs) generated by ChatGPT (an AI model) and two reputable hospitals, assessing AI's potential in producing high-quality patient information. Subjects/Methods 15 ophthalmologists and 32 patients evaluated three anonymised cataract PILs: one generated by ChatGPT, one from Mount Sinai Hospital (USA) and Manchester Royal Eye Hospital (UK). Doctors used the DISCERN tool (16 questions) for quality assessment. Patients used a shortened version (5 questions). Additional preference and readability questions were added, alongside a readability assessment. PIL ratings and differences between doctor and patient scores were compared. Results The ChatGPT PIL scored lowest amongst doctors (mean 42.75 (SD 9.06)/75), followed by Manchester (47.04 (8.56)/75), with Mount Sinai's PIL highest (54.65 (7.09)/75) ( p =<0.01). Patients similarly rated ChatGPT lowest (mean total score 4.50 (0.21)/5), with Manchester highest (4.84 (0.06)/5) ( p = 0.04). Despite this, doctors were evenly divided on their preferred PIL, while more patients preferred ChatGPT over Mount Sinai. Mount Sinai's PIL had the highest inter-rater reliability( k = 0.38, 95% CI 0.10–0.60), and ChatGPT the lowest ( k = 0.13, 95% CI 0.10–0.15). ChatGPT had the lowest Flesch Reading Ease score but doctors rated it most readable. Conclusions This study is the first to assess AI-generated cataract PILs using doctor and patient feedback. While ChatGPT received the lowest ratings, some favoured it, particularly for its clarity and readability. Doctors’ highest-rated PIL was the patients’ least favoured. This study highlights AIs potential in PIL development and the importance of doctor and patient feedback in this process.