Effectiveness of Chatbots in Mental Health Screening and Assessment: A Systematic Review (Preprint)

BACKGROUND Background: Chatbots, conversational agents, Large Language Model (LLM)-based systems are increasingly being applied to mental health screening and assessment. However, the evidence remains heterogeneous, and the clinical readiness of these tools for routine implementation is not clear. Objective: This systematic review aimed to evaluate the effectiveness, accuracy, reliability, acceptability, and feasibility of chatbots and AI-based conversational systems for mental health screening, structured assessment, diagnostic support, prediction, and risk stratification in adults. Methods: A systematic search was conducted across PubMed/MEDLINE, Embase, Scopus, Web of Science Core Collection, and APA PsycInfo for studies published between 2019 and April 2026, in English or Spanish. Eligible studies included adult populations and evaluated chatbots, conversational agents, or generative LLMs applied to mental health screening or assessment-related tasks. Study selection and data extraction were conducted independently by different reviewers. Risk of bias was assessed using QUADAS-2 and Joanna Briggs Institute checklists. Due to substantial clinical and methodological heterogeneity, findings were synthesized narratively and supported by descriptive quantitative summaries of reported performance metrics. Results: Twenty-three studies were included, covering depression, anxiety, psychological distress, suicide risk, cognitive impairment, schizophrenia-spectrum disorders, posttraumatic stress disorder, substance use, insomnia, burnout, and broader psychiatric symptom dimensions. Rule-based and structured chatbots showed acceptable reliability, feasibility, and user acceptability for administering validated screening instruments. LLM-based and natural language processing approaches showed promising task-specific performance in screening, symptom extraction, classification and prediction tasks, particularly when combined with machine-learning methods, electronic health record data, or clinical transcripts. Reported metrics included high internal consistency for chatbot-administered scales, AUC values commonly around 0.80-0.96 in selected models, and F1-scores of approximately 0.82-0.95 in some diagnostic-support and risk-detection tasks. However, performance varied substantially across conditions, reference standards, model architectures, and study designs. Risk-of-bias assessment identified recurrent concerns related to participant selection, small or convenience samples, retrospective designs, limited blinding, imperfect reference standards, lack of external validation, and incomplete reporting. Conclusions: Chatbots and AI-based conversational systems show potential as adjunctive tools for mental health screening, structured assessment, symptom extraction, risk stratification, and diagnostic-support tasks. Current evidence, however, is methodologically heterogeneous and does not support autonomous diagnosis or unsupervised clinical deployment. Future studies should prioritize prospective validation in representative clinical populations, standardized reporting of diagnostic accuracy metrics, external validation, assessment of algorithmic bias, and evaluation of safety, acceptability, and clinical utility within real-world care pathways. OBJECTIVE Mental health disorders constitute one of the leading causes of disability worldwide, affecting 970 million people in 2019, with depressive and anxiety disorders being the most prevalent diagnoses [1]. Globally, these conditions account for approximately 14% of disability-adjusted life years (DALYs), with depression alone ranking as the third leading cause of disease burden [2]. In Spain, it is estimated that one in four adults will experience a mental health disorder throughout their lifetime, reflecting a significant burden on the healthcare system [3]. However, more than 50% of individuals with symptoms related to these disorders do not access timely evaluation due to structural barriers, social stigma, and a shortage of professionals [4]. Average waiting times for initial psychiatric assessment can exceed 60 days in many European health systems, with significant geographical disparities between urban and rural areas [5]. This treatment gap is particularly pronounced in low- and middle-income countries, where up to 85% of individuals with mental health needs receive no care at all [6]. This care gap, exacerbated by prolonged waiting lists and geographical inequalities, has driven the search for scalable digital solutions that can expand coverage and alleviate clinical care services. In this context, chatbots and conversational agents based on artificial intelligence (AI) emerge as promising tools for psychological screening and assessment, offering continuous availability (24/7), replicating validated protocols, and automatically referring urgent cases to professionals [7,8]. Early systematic reviews demonstrated that conversational agents show promising efficacy and acceptability in mental health contexts, though most evidence focused on therapeutic interventions rather than assessment capabilities [9,10]. These systems can be broadly categorized into rule-based chatbots, which follow predetermined decision trees and administer standardized psychometric scales, and generative models based on large language models (LLMs) such as ChatGPT-4o, Claude 3.5, and Gemini 1.5, which can engage in more flexible, context-aware conversations and interpret unstructured clinical narratives [11,12]. Rule-based systems, which dominated mental health chatbot research until 2023, offer advantages in transparency, clinical control, and consistent administration of validated instruments, with proven reliability for structured screening protocols [10,13]. However, they lack the flexibility to adapt to individual patient narratives or handle complex, nuanced clinical presentations [14]. Furthermore, the integration of prioritization algorithms and real-time data analysis enables the generation of more accurate and personalized assessments, aligning with the European Commission's recommendations for the digitalization of mental healthcare [4]. The landscape shifted dramatically with the emergence of advanced LLMs from 2023 onward. Recent systematic analyses show that LLM-based chatbots surged from 16% of studies in 2022 to 45% in 2024, becoming the most frequently studied architecture in mental health AI research [15]. These models demonstrate capabilities that extend beyond simple symptom checklists: they can engage in open-ended clinical dialogue, interpret contextual information, generate psychodynamic formulations, and perform diagnostic reasoning that approaches the performance of experienced clinicians [8,11]. For instance, GPT-4 has passed psychiatric licensing examinations and demonstrated diagnostic accuracy comparable to psychiatrists in standardized clinical scenarios [16,17]. Recent evaluations of state-of-the-art models, including DeepSeek-R1, GPT-4.1, and Llama 4, have shown exceptional performance in both mental health knowledge assessment and illness diagnosis tasks, which evidences the technical capacity of the models [18]. In recent years, AI has demonstrated a disruptive role in mental health, with applications ranging from early symptom detection to the automation of therapeutic interventions. Recent reviews indicate that conversational systems can accurately identify linguistic and behavioral patterns associated with depression and anxiety in screening contexts, with high precision rates [7,19]. Beyond depression and anxiety, emerging evidence suggests potential applications in detecting cognitive impairment [12,20], assessing suicide risk [21,22], evaluating thought disorder in schizophrenia [23,24], and screening for postpartum PTSD [25]. A detailed scoping review of 95 studies found that 71% of LLM applications in mental health focused on screening and detection tasks, with reported accuracies ranging from 0.80 to 0.96 for depression and anxiety classification [7]. In suicide risk assessment, LLMs have demonstrated correlations with expert clinicians exceeding r = 0.90 in standardized vignette studies; however, concerns persist about the potential underestimation of risk in complex cases [26,27]. These technologies not only expand access but also facilitate personalized, scalable, and cost-effective interventions, which is critical given the growing demand and shortage of human resources. However, their implementation raises ethical and methodological challenges, including the need for clinical supervision, data privacy protection, algorithmic transparency, and absence of longitudinal evidence on sustained impact [28,29]. Algorithmic bias is a particular concern: AI systems for mental health have demonstrated differential performance across demographic subgroups and may encode biases related to race, gender, age, and socioeconomic context that risk perpetuating existing health disparities [28,30–33]. For example, qualitative analyses of leading LLMs found that race-explicit or race-implied patient information frequently resulted in inferior treatment recommendations, though diagnostic decisions showed less bias [7]. Moreover, current evidence remains heterogeneous and limited, focusing primarily on acceptability and user experience, while diagnostic validity, clinical utility, reliability, and equity in access have been less systematically explored. In fact, previous systematic reviews have examined AI applications in mental health intervention and user acceptability [7,19], but these studies largely predate the emergence of advanced generative models and do not specifically focus on assessment capabilities. Despite growing interest in the use of chatbots and AI systems for psychological assessment, avai