Assessing the Performance of ChatGPT in Correctly Evaluating Preventive Screening Clinical Vignettes Commonly Seen in Obstetrics and Gynecology Practice [ID 2683617]

INTRODUCTION: Preventive screening is vital in obstetrics and gynecology practice, where the accurate assessment of clinical vignettes is pivotal for delivering optimal care. This research delves into ChatGPT’s performance, an advanced language model, in evaluating common obstetrics and gynecology preventive screening scenarios. METHODS: We compiled a diverse dataset of obstetrics and gynecology clinical vignettes, covering areas like cervical and breast cancer screening, sexually transmitted infection testing, and prenatal care. Using 20 vignettes initially generated by ChatGPT and enriched with specific details, we rigorously assessed ChatGPT’s responses for clinical accuracy and relevance. Our evaluation was guided by quantitative metrics, aligning ChatGPT’s performance with recognized guidelines such as United States Preventive Services Task Force, American Society for Colposcopy and Cervical Pathology, American College of Obstetricians and Gynecologists, and more. RESULTS: ChatGPT displayed promise in comprehending obstetrics and gynecology clinical information. Out of 20 vignettes, only 2 (10%) fell short of evaluating the clinical vignette adequately. Among the remaining 18 vignettes, an impressive 15 (83%) adhered to the latest clinical guidelines, whereas 3 (17%) evaluations proved inaccurate. Notably, in 19 out of 20 vignettes (95%), ChatGPT deferred to the provider’s clinical judgment. CONCLUSION: This study underscores ChatGPT’s potential to assist obstetrics and gynecology practitioners in preventive screening decision-making. While promising, ChatGPT should complement, not replace, human expertise. Limitations include a focus on quantitative metrics; future research should delve into qualitative aspects of ChatGPT’s performance. Integrating ChatGPT into obstetrics and gynecology practice could enhance preventive screening quality and efficiency, potentially improving women’s health care outcomes. This research contributes to the broader discourse on AI integration in health care, particularly in women’s preventive health care, paving the way for further AI-driven solutions.