Evaluating the Efficacy of ChatGPT-3.5 Versus Human-Delivered Text-Based Cognitive-Behavioral Therapy: A Comparative Pilot Study

OBJECTIVE: The aim of this study was to compare the effectiveness of an artificial intelligence (AI) therapist (ChatGPT-3.5) and a human therapist in delivering text-based cognitive-behavioral therapy (CBT). Cognitive Therapy Rating Scale (CTRS) scores and qualitative responses from participants were used to assess ChatGPT-3.5's capabilities within the limitations of this deployment. METHODS: A cross-sectional survey was administered to 208 mental health professionals and trainees, 75 of whom completed the study. Participants assessed two text-based CBT transcripts (AI therapist and human therapist) using the CTRS and provided qualitative feedback. Both the AI therapist and the human therapist were presented with identical clinical scenarios to ensure consistency. RESULTS: The human therapist outperformed ChatGPT-3.5 across most CTRS domains (α=0.001). More than half (52%) of the respondents rated the human therapist's agenda setting highest, with a score of 6, compared with 28% who gave this rating to ChatGPT-3.5. In elicited feedback, 29% rated the human therapist as highly effective (score=6), whereas only 9% rated ChatGPT-3.5 similarly (p<0.001). The human therapist also scored higher on guided discovery (24% vs. 12%, p<0.001). ChatGPT-3.5 was rated similarly to the human therapist in understanding the patient's internal reality (36% vs. 19%, p=0.004) but was often seen as less personalized and more rigid. CONCLUSIONS: The findings suggest that although ChatGPT-3.5 may complement human-based therapy, this specific implementation of AI lacked the depth required for stand-alone use. These findings, however, cannot be generalized to all AI-based therapy because the study did not account for the diverse ways therapy can be delivered.