Digitally Assisted Clinical Decision-Making in Traditional Chinese Medicine: Comparative Study of 5 Large Language Models

Background: Traditional Chinese medicine (TCM) clinical decision-making involves complex integration of syndrome differentiation, constitutional assessment, and individualized treatment selection, creating challenges for standardization and quality assurance. While large language models (LLMs) demonstrate capabilities in medical knowledge integration and clinical reasoning, their application to TCM remains largely unexplored, particularly regarding syndrome differentiation principles and prescription formulation. Objective: This study evaluated 5 contemporary LLMs in TCM clinical decision-making and assessed human-artificial intelligence (AI) collaboration compared with independent approaches. Specific objectives were to benchmark LLM performance in TCM knowledge assessment, evaluate clinical case analysis capabilities, identify the optimal model, and assess the quality, efficiency, and acceptability of human-AI collaboration. Methods: In total, 5 mainstream LLMs were evaluated-Claude 3.7 Sonnet-Extended (Anthropic), ChatGPT 4.5 (OpenAI), Grok3-DeepSearch (xAI), Gemini 2.0 Flash Thinking Experimental (Google), and DeepSeek-R1 (DeepSeek). The evaluation consisted of four phases, (1) TCM knowledge assessment using 160 standardized questions, (2) clinical case analysis of 30 cases representing different disease systems and complexity levels, (3) optimal model selection using weighted scoring (40% knowledge and 60% clinical analysis), and (4) clinical application assessment involving 10 TCM practitioners and 2 experts comparing physician-only, AI-only, and human-AI collaboration across 5 clinical cases. Statistical analysis included descriptive statistics, reliability analysis, comparative testing, and regression analysis. Results: DeepSeek-R1 demonstrated superior performance across both evaluation domains, achieving 96.7% accuracy in knowledge assessment and 17.31/20 (SD 2.65) in clinical case analysis, significantly outperforming other models (P<.001). Human-AI collaboration achieved significant improvements compared with physician-only decision-making, with 16.1% quality enhancement (33.62 vs 28.97; P<.001) and 66.1% time reduction (162.6 s vs 479.2 s; P<.001). System usability was rated favorably (System Usability Scale score=76.8; P=.002), with high acceptance rates (74.25% adoption, 24% modification, and 1.75% rejection). AI assistance provided the greatest benefits in prescription formulation and medication selection (P<.001). Conclusions: LLMs, particularly DeepSeek-R1, demonstrate substantial capabilities in TCM knowledge assessment and clinical case analysis. Human-AI collaboration significantly enhanced clinical decision-making quality and efficiency while maintaining high physician acceptance. These findings provide compelling evidence for the clinical value of AI-assisted decision-making in TCM, suggesting potential solutions to current challenges in knowledge standardization, clinical training, and health care delivery efficiency. Strategic implementation of AI assistance could significantly enhance the quality, efficiency, and accessibility of TCM care while preserving fundamental principles of individualized treatment.