Clinical accuracy, safety, triage, and test-retest consistency of general-purpose large language models for patient-centred respiratory questions: a blinded paired benchmarking study

急诊分诊台 医学 标杆管理 一致性(知识库) 预警得分 梅德林 危害 重症监护医学 患者安全 初级保健 急诊医学 医疗急救 呼吸系统 医学物理学 临床试验 医疗保健
作者
Wei Xiao,廖凌飞,Xiaoyi Chen,Jun Cai,駱旭琛,Jiehua Deng
出处
期刊:Frontiers in Medicine [Frontiers Media]
卷期号:13
标识
DOI:10.3389/fmed.2026.1940881
摘要

Background General-purpose large language models are increasingly used by patients to obtain health information and guidance on whether medical care is required. In respiratory medicine, clinically plausible but incomplete responses may still cause harm if warning signs are missed or urgent assessment is delayed. This study compared the clinical accuracy, safety, triage performance, and test-retest consistency of three contemporary large language models when answering patient-centred respiratory questions. Methods A blinded, paired, repeated-response benchmarking study was conducted using 150 respiratory questions covering seven clinical domains and equally distributed across low-, moderate-, and high-risk strata. Each question was submitted twice in separate conversations to ChatGPT-5.5, Gemini 3.5 Flash, and Grok 4.1, generating 900 responses. Two respiratory physicians independently assessed anonymised responses against prespecified item-specific reference standards, with prespecified disagreements adjudicated by a third senior respiratory physician. The primary outcome was the proportion of clinically acceptable responses, defined using accuracy, safety, warning-sign recognition, and triage criteria. Secondary outcomes included major safety errors, under-triage, safety-netting, readability, and test-retest consistency. Results In the first response round, clinically acceptable responses were generated for 88.0% of questions by ChatGPT-5.5, 81.3% by Gemini 3.5 Flash, and 74.0% by Grok 4.1 (overall P < 0.001). Major safety errors occurred in 3.3%, 6.0%, and 10.0% of responses, respectively ( P = 0.009). Under-triage occurred in 4.7% of ChatGPT-5.5 responses, 9.3% of Gemini 3.5 Flash responses, and 13.3% of Grok 4.1 responses. Among high-risk questions, exact or acceptable triage agreement was 84.0%, 72.0%, and 62.0%, while under-triage occurred in 8.0%, 16.0%, and 22.0%, respectively. Test-retest clinical response consistency was highest for ChatGPT-5.5 (94.0%), followed by Gemini 3.5 Flash (89.3%) and Grok 4.1 (84.7%). Conclusions The three models answered many respiratory questions accurately and safely, but clinically important differences were observed in safety, triage, and response consistency. All models produced under-triage and major safety errors, particularly in high-risk respiratory scenarios. These systems may support selected low-risk educational tasks but should not independently determine emergency triage, medication changes, or oxygen adjustment.

科研通智能强力驱动
Strongly Powered by AbleSci AI
科研通是完全免费的文献互助平台,具备全网最快的应助速度,最高的求助完成率。 对每一个文献求助,科研通都将尽心尽力,给求助人一个满意的交代。
实时播报
zzk发布了新的文献求助10
刚刚
04d的应助被luchong采纳,获得10
刚刚
坚定的珊珊完成签到 ,获得积分10
刚刚
曾宪煌发布了新的文献求助10
刚刚
1秒前
华仔的应助被dbyte采纳,获得10
1秒前
世外仙姝发布了新的文献求助10
2秒前
Lily发布了新的文献求助10
2秒前
xm完成签到,获得积分10
2秒前
2秒前
2秒前
2秒前
清辉影发布了新的文献求助10
2秒前
稳健发布了新的文献求助30
3秒前
lllliiii发布了新的文献求助10
4秒前
田様的应助被corleeang采纳,获得10
4秒前
zqxu完成签到,获得积分10
4秒前
思源的应助被lbm采纳,获得10
5秒前
5秒前
5秒前
贪玩的筝发布了新的文献求助10
6秒前
坚持自我发布了新的文献求助10
6秒前
6秒前
6秒前
可爱的函函的应助被ww采纳,获得10
6秒前
棠xx完成签到,获得积分10
7秒前
研友_VZG7GZ的应助被wujx采纳,获得10
7秒前
汉堡包的应助被核桃采纳,获得10
7秒前
FashionBoy的应助被核桃采纳,获得10
7秒前
悠悠完成签到,获得积分10
7秒前
xinyuanjia的应助被核桃采纳,获得10
7秒前
牧百川发布了新的文献求助10
7秒前
wanci的应助被核桃采纳,获得10
8秒前
momo完成签到 ,获得积分10
8秒前
慕青的应助被核桃采纳,获得10
8秒前
wocc的应助被核桃采纳,获得10
8秒前
科研通AI6.3的应助被核桃采纳,获得10
8秒前
852的应助被核桃采纳,获得10
8秒前
世外仙姝完成签到,获得积分10
8秒前
9秒前
高分求助中
(应助此贴封号)【重要!!请各用户(尤其是新用户)详细阅读】【科研通的精品贴汇总】 10000
拉伸高分子熔体中缠结行为的模拟和小角中子散射研究 1700
The World Atlas of Wine 9th edition 1000
奈特基础组织学:配套组织病理学(第4版)(Netter's Essential Histology With Correlated Histopathology) 800
八旗制度の研究 600
Improving the wear resistance of vehicle gear oil by oil soluble LaF3 nanoparticles and typical additives of gear oil 500
Boundaries and Bodhisattvas 500
热门求助领域 (近24小时)
化学 材料科学 医学 生物 纳米技术 计算机科学 工程类 有机化学 化学工程 生物化学 物理 复合材料 内科学 催化作用 光电子学 人工智能 细胞生物学 无机化学 物理化学 数学
热门帖子
关注 科研通微信公众号,转发送积分 8183142
求助须知:如何正确求助?哪些是违规求助? 9572420
关于积分的说明 21825253
捐赠科研通 7779922
什么是DOI,文献DOI怎么找? 3433850
关于科研通互助平台的介绍 2558060
邀请新用户注册赠送积分活动 2480124