Evaluation of ChatGPT ‐4 responses to glaucoma patients' questions: Can artificial intelligence become a trusted advisor between doctor and patient?

Glaucoma is a leading cause of irreversible blindness worldwide.1 Managing glaucoma is challenging, and some patients may still experience vision loss despite receiving treatment.2 Because of its insidious progression and its capacity to profoundly affect the daily lives and routines of patients despite treatment, this disease can evolve into a perpetual challenge for afflicted individuals. Patients often face difficulty accessing information about treatment methods, prognosis, and medication side effects due to reasons such as insufficient time in clinical settings or uncertainty about how to formulate their questions. ChatGPT is a sophisticated language model created by OpenAI that can produce text in response to user prompts.3 Numerous research studies have demonstrated the efficacy of artificial intelligence models in the field of ophthalmology.4 Searching for information about health and disease online is a natural part of coping with and living with illness.5 This research involved reaching a consensus among two ophthalmologists regarding frequently encountered questions, collecting responses from ChatGPT to these questions, and subsequently evaluating the accuracy of these responses.50 questions frequently asked to physicians by glaucoma patients in our clinic and from the websites of the American Academy of Ophthalmology* and the American Glaucoma Association* were identified. Fifty questions divided into five categories. These categories were determined as general questions/aetiology, prognosis, treatment/prevention, side effects of medication, and surgery. All the questions are shown in the Table 1. Two experienced ophthalmologists evaluated each chatbot response for accuracy, satisfaction with the quality of responses for real patient use, and the potential harm the responses could pose to real patients. To evaluate the response generated by the chatbots, a 5-point Likert scale was used Table 2. All ophthalmologists were blinded to the chatbot source. Our results indicate a high level of concordance between ChatGPT-4 predictions and evaluations, with Cronbach's alpha analysis (Cronbach's alpha: 0.841; 95% confidence interval). The distribution of the scores given by two ophthalmologists to the answers to the questions is shown in Table 3. The scores ranged from an average of 4.8 ± 0.4 for the highest-rated question to an average of 3.1 ± 0.3 for the lowest-rated question. The lowest accuracy scores were obtained from questions related to medication side effects and prognosis, while the highest accuracy scores were from questions related to treatment and prevention. Categorization of Questions and Ratings of Responses Generated by ChatGPT are shown Table 4. Incomplete or entirely irrelevant responses, as well as responses containing significant inaccuracies with the potential to cause harm, were not observed. All responses were related to the topic, but varied in the degree of accuracy. Thirty responses (60%) were rated as containing no inaccuracies by both ophthalmologists. In the remaining 16 responses (32%), one or both doctors identified minor inaccuracies that did not have the potential to cause harm. In four responses (8%), both doctors identified major inaccuracies that did not have the potential to cause harm. All responses were relevant to the topic, and there were no entirely irrelevant responses, significant inaccuracies, or responses with the potential to cause harm observed. In this study, the accuracy of ChatGPT's responses to typical patient questions related to glaucoma was investigated. Overall, ChatGPT provided relevant responses to all questions. The accuracy of the responses was generally high, and no significant inaccuracies with the potential to cause harm were observed. Notably, ChatGPT's ability to deliver instant responses and potentially alleviate patient anxieties underscores its utility in promptly addressing medical concerns. However, it is crucial to emphasise the ongoing necessity of professional medical oversight. Considering the immediacy of ChatGPT's responses compared to the availability of ophthalmologists, patients are likely to resort to ChatGPT or similar AI services for information. As artificial intelligence becomes increasingly integrated into ophthalmology, the challenge lies in effectively incorporating these technologies to enhance patient care while maintaining stringent standards for patient safety and well-being. None. The authors declare no conflicts of interest. The data that support the findings of this study are available from the corresponding author upon reasonable request.