MP28-17 THE QUALITY OF CHATBOT RESPONSES RELATED TO PEYRONIE’S DISEASE

You have accessJournal of UrologySexual Function/Dysfunction: Peyronie's Disease (MP28)1 May 2024MP28-17 THE QUALITY OF CHATBOT RESPONSES RELATED TO PEYRONIE'S DISEASE Christopher J. Warren, Victoria S. Edmonds, Nicolette G. Payne, Sarah Y. Wu, JennaKay Colquitt, and Nahid Punjani Christopher J. WarrenChristopher J. Warren , Victoria S. EdmondsVictoria S. Edmonds , Nicolette G. PayneNicolette G. Payne , Sarah Y. WuSarah Y. Wu , JennaKay ColquittJennaKay Colquitt , and Nahid PunjaniNahid Punjani View All Author Informationhttps://doi.org/10.1097/01.JU.0001008872.42208.7a.17AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: Large language model (LLM) Chatbots, a form of artificial intelligence (AI) that rely on user prompts to mimic conversation have been shown to excel at many tasks in the medical field. Our aim was to assess the information generated from 4 LLMs with searches related to Peyronie's disease (PD), to improve responses, and to assess responses to artificial patient messages (Figures 1,2). METHODS: The National Institute of Health's (NIH) frequently asked questions related to PD were entered into 4 LLMs unprompted and prompted (Figures 1, 2). The responses were evaluated for overall quality using the DISCERN questionnaire (Figure 1). Accuracy and completeness of LLM responses to 11 pre-surgical patient messages were evaluated with previously accepted Likert scales (Figure 2). Descriptive statistics and analysis were performed. RESULTS: Without prompting, the quality of information was moderate across all LLMs but improved to high quality with prompting (Table 1). LLMs were accurate and complete with an average score of 5.8 out of 6.0 (SD 0.5) and 3.0 out of 3.0 (SD 0.2) respectively. The average Flesch-Kincaid reading level was grade 11.7 (SD 2.1). Chatbots were unable to communicate at a grade 8 reading level when prompted. In contrast, the reading level of the NIH website was significantly lower (9.8, SD 2.1, p<.05) than the prompted LLM responses. CONCLUSIONS: LLMs may become a valuable tool for patient education for PD but they currently rely on clinical context and appropriate prompting by humans to be useful. Unfortunately, their prerequisite reading level remains higher than that of the average patient. Given their increasing uptake, patients and physicians should be educated on how to interact with these LLMs to elicit the most appropriate responses. In the future, LLMs may reduce burnout by helping physicians respond to patient messages. Download PPTDownload PPT Source of Funding: No funding was provided for this research © 2024 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 211Issue 5SMay 2024Page: e479 Advertisement Copyright & Permissions© 2024 by American Urological Association Education and Research, Inc.Metrics Author Information Christopher J. Warren More articles by this author Victoria S. Edmonds More articles by this author Nicolette G. Payne More articles by this author Sarah Y. Wu More articles by this author JennaKay Colquitt More articles by this author Nahid Punjani More articles by this author Expand All Advertisement PDF downloadLoading ...