Chatbot Performance in Defining and Differentiating Palliative Care, Supportive Care, Hospice Care

Abstract

Context

: Artificial intelligence (AI) chatbot platforms are increasingly used by patients as sources of information. However, there is limited data on the performance of these platforms, especially regarding palliative care terms.

Objectives

: We evaluated the accuracy, comprehensiveness, reliability, and readability of three AI platforms in defining and differentiating "palliative care," "supportive care," and "hospice care."

Methods

: We asked ChatGPT, Microsoft Bing Chat, Google Bard to define and differentiate "palliative care," "supportive care," and "hospice care" and provide three references. Outputs were randomized and assessed by six blinded palliative care physicians using 0-10 scales (10=best) for accuracy, comprehensiveness, and reliability. Readability was assessed using Flesch Kincaid Grade Level and Flesch Reading Ease scores.

Results

: The mean (SD) accuracy scores for ChatGPT, Bard, and Bing Chat were 9.1 (1.3), 8.7 (1.5), and 8.2 (1.7), respectively; for comprehensiveness, the scores for the three platforms were 8.7 (1.5), 8.1 (1.9), and 5.6 (2.0), respectively; for reliability, the scores were 6.3 (2.5), 3.2 (3.1), and 7.1 (2.4), respectively. Despite generally high accuracy, we identified some major errors (e.g. Bard stated that supportive care had "the goal of prolonging life or even achieving a cure"). We found several major omissions, particularly with Bing Chat (e.g. no mention of interdisciplinary teams in palliative care or hospice care). References were often unreliable. Readability scores did not meet recommended levels for patient educational materials.

Conclusion

: We identified important concerns regarding the accuracy, comprehensiveness, reliability, and readability of outputs from AI platforms. Further research is needed to improve their performance.