Comparative assessment of patient education content generated by five large language models on post-stroke complications: A cross-sectional study of safety, accuracy, reliability, readability, and empathy

作者
Long-Xian Liu,Huanhuan Wang,Shuang Lu,Guo Yu,Jia-Fen Li,Qing-Hua Lai,Ben Guo,Miao Dong,Lili Zhang,Mei-Xia Ye,Meng Wu
出处
期刊:Digital health [SAGE Publishing]
卷期号:12
标识
DOI:10.1177/20552076261493266
摘要

Objective To compare the safety, accuracy, reliability, readability, and empathy of patient education content generated by five large language models (LLMs) for post-stroke complications. Methods This CHART-guided cross-sectional comparative study evaluated ChatGPT, Gemini, DeepSeek, Doubao, and Qwen using 40 standardized patient- and caregiver-facing prompts. Each model generated one response per prompt in an independent session. Three trained raters—a neurologist and two nursing postgraduate researchers specializing in neurological and stroke rehabilitation nursing—independently assessed safety, accuracy, empathy, and information quality using predefined criteria, including DISCERN, EQIP, GQS, and JAMA. Readability was assessed using six established indices. Agreement was evaluated using Fleiss’ kappa and ICC(2,1). Model comparisons used Cochran’s Q test for safety and Friedman tests with Kendall’s W for non-binary outcomes, followed by Benjamini–Hochberg-adjusted pairwise analyses where appropriate. Results The final dataset comprised 2,600 metric-level observations. Inter-rater agreement was acceptable to high (Fleiss’ kappa = 0.740; ICCs = 0.838–0.893). Safe-response rates were high (95.0%–97.5%) and did not differ significantly ( Q = 0.750, P = 0.945). Significant between-model differences were found for accuracy, empathy, DISCERN, EQIP, GQS, and all readability indices (all P < 0.001), although the overall effect for accuracy was modest ( W = 0.230). JAMA scores were uniformly low and showed no significant difference ( P = 0.888), indicating limited transparency in standalone chatbot responses. No model consistently met the recommended sixth-grade reading level or achieved mean Flesch Reading Ease ≥80. Conclusions The evaluated LLMs generated mostly safe and clinically relevant English-language responses, but performance varied across domains and no model was consistently superior. LLMs may assist in drafting patient education materials, but clinical review, evidence verification, source attribution, and readability optimization remain necessary before patient-facing use. Findings are specific to the prompts and single-generation outputs evaluated.

科研通智能强力驱动
Strongly Powered by AbleSci AI
科研通是完全免费的文献互助平台,具备全网最快的应助速度,最高的求助完成率。 对每一个文献求助,科研通都将尽心尽力,给求助人一个满意的交代。
实时播报
muyouwifi发布了新的文献求助10
刚刚
FashionBoy的应助被优美的未来采纳,获得10
1秒前
小蘑菇的应助被111采纳,获得10
1秒前
1秒前
Hello的应助被一杯芝士采纳,获得10
1秒前
今后的应助被CQNU破产之人采纳,获得10
2秒前
Orange的应助被szxa采纳,获得10
2秒前
Free完成签到,获得积分20
2秒前
sx发布了新的文献求助10
2秒前
小宋完成签到 ,获得积分10
2秒前
纯真大象发布了新的文献求助10
4秒前
zxd发布了新的文献求助10
4秒前
5秒前
会相遇完成签到,获得积分10
5秒前
xyl完成签到,获得积分10
5秒前
549发布了新的文献求助10
5秒前
6秒前
HYYY完成签到,获得积分10
6秒前
firework完成签到,获得积分10
7秒前
Qiiii完成签到,获得积分10
8秒前
科研通AI6.3的应助被阿鹏采纳,获得10
10秒前
WU_JIABIN发布了新的文献求助10
11秒前
科研通AI6.2的应助被Liy采纳,获得10
13秒前
单薄幻梦发布了新的文献求助10
13秒前
13秒前
14秒前
田様的应助被无敌小宝贝采纳,获得10
14秒前
夏安完成签到,获得积分10
15秒前
15秒前
qin202569发布了新的文献求助10
16秒前
CodeCraft的应助被纯真大象采纳,获得10
16秒前
16秒前
17秒前
二娃发布了新的文献求助10
17秒前
南瓜完成签到,获得积分10
17秒前
努力小人完成签到,获得积分10
17秒前
NexusExplorer的应助被NPC-CBI采纳,获得10
18秒前
开心盼海发布了新的文献求助10
19秒前
19秒前
20秒前
高分求助中
(应助此贴封号)【重要!!请各用户(尤其是新用户)详细阅读】【科研通的精品贴汇总】 10000
拉伸高分子熔体中缠结行为的模拟和小角中子散射研究 1700
Grapes 1000
The World Atlas of Wine 9th edition 1000
奈特基础组织学:配套组织病理学(第4版)(Netter's Essential Histology With Correlated Histopathology) 800
John McDowells Bildungstheorie 800
八旗制度の研究 600
热门求助领域 (近24小时)
化学 材料科学 医学 生物 纳米技术 计算机科学 工程类 有机化学 化学工程 生物化学 物理 复合材料 内科学 催化作用 光电子学 人工智能 细胞生物学 无机化学 物理化学 数学
热门帖子
关注 科研通微信公众号,转发送积分 8178960
求助须知:如何正确求助?哪些是违规求助? 9569638
关于积分的说明 21805682
捐赠科研通 7774071
什么是DOI,文献DOI怎么找? 3432955
关于科研通互助平台的介绍 2556897
邀请新用户注册赠送积分活动 2478640