Mind AI's Mind: A Clinically Aligned Explainable AI Pipeline for Depression Diagnosis via Large Language Models

The rise of artificial intelligence (AI) in medical diagnostics has highlighted an essential need for transparent and interpretable systems, particularly in the field of mental health. The opaque decision-making in “black-box” AI models creates a challenge in clinical settings: they risk losing trust or causing uncritical reliance. This paradox jeopardizes mental healthcare, where skepticism or unwarranted confidence in AI can harm patient care. Effective explainability mechanisms are therefore essential not only to earn trust but to support responsible use by allowing clinicians to critically assess and verify AI-driven outputs. Thus, we propose a novel explainable AI (XAI) pipeline for automated depression diagnosis, designed to integrate both system-level and human-level interpretability. This dual approach is vital in clinical settings, as it combines rigorous statistical validation with clear, actionable insights that enhance practitioner confidence in AI-generated diagnoses. The pipeline leverages deep learning models for classification, augmented by traditional system-level XAI techniques and a Retrieval-Augmented Generation (RAG)-enhanced Large Language Model (LLM). With the integration of LLMs, the system translates abstract system-level explanations into understandable, natural language narratives. This provides a crucial cross-verification step that fosters calibrated trust: it mitigates the dual risks of under-trust, by providing a clear rationale, and over-trust, by flagging diagnostic inconsistencies within the AI models. This capability is critical for ensuring both user trust and satisfaction, as it empowers practitioners to critically assess and validate AI-driven insights. Through comprehensive human evaluations conducted by medical professionals, this approach demonstrates high alignment with clinical diagnostic indicators, underscoring the value of combining system-level and human-level explanations to make complex AI processes transparent and clinically meaningful. Validated across three diverse datasets-KangNing, EDAIC-WOZ, and CALLM-each presenting unique structural challenges from structured clinical inquiries to narrative-driven dialogues, our method bridges the gap between technical AI outputs and practitioner understanding, marking a significant advancement toward a trust-based, widely adoptable AI diagnostic tool in mental health care. By introducing a comprehensive framework for combining statistical rigor with narrative clarity, this work represents a critical step toward closing the gap between black-box AI systems and real-world clinical adoption. While our study is limited to depression diagnosis, the proposed framework illustrates a pathway toward explainable clinical AI systems. Future work may explore its adaptability to other medical contexts.