Optimizing ChatGPT

Introduction: Patients rely on online searches for patient education materials (PEMs). PEMs are recommended to be written at or below a sixth-grade reading level but are regularly written at a college reading level. Using prompt engineering, we assess the information, misinformation, and readability of ChatGPT responses to urologic oncology questions. Methods: Forty-five questions relating to prostate, bladder, and kidney cancer were presented to ChatGPT (version 4o, OpenAI). Quality of health information was assessed using DISCERN (1 [low] to 5 [high]). Understandability and actionability were assessed using PEMAT-P (0 [low] -100% [high]). Misinformation was scored from 1 [no misinformation] to 5 [high misinformation]. Grade and reading level were calculated using the Flesch-Kincaid scale [5 (easy) to 16 (difficult), and 100-90 (5th grade level) to 10-0 (professional level), respectively]. Prompt engineering was then applied to responses and evaluated. Results: ChatGPT answers are highly accurate but too advanced of a reading level and lacked explanations of benefits, risks, visual aids, actionability, and citations. Using prompt engineering, DISCERN (3.42-4.47, p<0.0001), PEMAT-P understandability (88.4-95.5%, p<0.0001), and actionability (25.6-84.2%, p<0.0001), grade reading level (10.55.3, p<0.0001), and reading level (42 [college level] to 71.7 [7th grade], p<0.0001), all significantly improved. Misinformation did not change significantly. Conclusions: Using prompt engineering, ChatGPT provides highly accurate and understandable PEMs at a patient appropriate reading level and provides concrete resources for patient action. Urologists should understand prompt engineering and be involved in the development of artificial chatbots to optimize results.