Error Detection in Emergency Radiology Reports Using a Large Language Model: Multistage Evaluation Study

计算机科学 医学物理学 医学 医疗急救 放射科 梅德林 远程放射学 医疗保健 人工智能 应急响应
作者
Hui Shen,Tianyang Wu,Fei Wang,Jin Fang,Yuange Li,Xiaoling Wu,Shuyi Liu,Li‐Ting Chen,Qiuping Ren,Xiao Meng,Jiatong Xu,Jiatong Xu,Yujie Zhao,Xin Liu,Liaoyuan Wang,Guipeng Mai,Jingjing You,Zhe Jin,Xuewei Wu,Wenle He
出处
期刊:Journal of Medical Internet Research [JMIR Publications]
卷期号:28: e86841-e86841
标识
DOI:10.2196/86841
摘要

Background: Emergency radiology requires highly accurate reporting under time constraints; yet, increasing workloads raise the risk of errors. While large language models (LLMs) show potential for proofreading in general radiology, their performance in emergency settings and non-English contexts remains unclear. Objective: We aim to evaluate the performance of a domain-optimized LLM, DeepSeek-R1, for identifying errors in Chinese emergency radiology reports, with comparison against assessments by board-certified radiologists. Methods: We compiled 7435 emergency reports (dataset 1; radiography, computed tomography, and magnetic resonance imaging) collected from November 2024 to April 2025. In stage 1, a total of 5 LLMs were evaluated using 200 reports. The best model, DeepSeek-R1, proceeded to stages 2 and 3, where 0-shot and few-shot learning were tested on a separate set (n=100). Model performance was compared against 12 radiologists. Stage 4 validated real-world utility on 800 verified reports. Results: In subdataset 1, under stress-testing conditions, DeepSeek-R1 achieved a higher error detection rate in the few-shot setting than in the 0-shot setting (84.4% vs 60.9%, P=.003). Its performance exceeded that of radiology residents (84.4% vs 51.6% and 53.1%, respectively; both P<.05) and showed no statistically significant difference compared with senior radiologists and attending radiologists (84.4% vs 68.8%-93.8%, P=.26 to ≥.99). Compared with residents, DeepSeek-R1 detected more critical omissions (100% vs 25% and 50%; both P<.05) and other errors (92% vs 33% and 33%; both P=.02). In dataset 2, collected from independent institutions, DeepSeek-R1 achieved a detection rate of 95% under the few-shot setting. Reading time was shorter than that of human readers (92 vs 109 s). In real-world validation, DeepSeek-R1 identified 117 true reporting errors, yielding a positive predictive value of 56.5%. Conclusions: DeepSeek-R1 holds promise for improving quality control in emergency radiology reports. Its performance and efficiency support its use as an assistive proofreading tool in real-world radiology workflows.
最长约 10秒,即可获得该文献文件

科研通智能强力驱动
Strongly Powered by AbleSci AI
科研通是完全免费的文献互助平台,具备全网最快的应助速度,最高的求助完成率。 对每一个文献求助,科研通都将尽心尽力,给求助人一个满意的交代。
实时播报
搜集达人的应助被mm采纳,获得10
1秒前
吴123完成签到,获得积分10
3秒前
唐唐完成签到,获得积分10
4秒前
CYLiu完成签到,获得积分10
4秒前
刘湘龙完成签到,获得积分10
4秒前
顺心秋凌完成签到,获得积分10
5秒前
无花果的应助被none采纳,获得50
5秒前
6秒前
liugm完成签到,获得积分10
6秒前
Scarlett完成签到,获得积分10
6秒前
6秒前
7秒前
小二郎的应助被CYLiu采纳,获得10
7秒前
Hello的应助被红豆鲫鱼饼采纳,获得10
8秒前
星辰大海的应助被吴123采纳,获得10
8秒前
8秒前
8秒前
8秒前
9秒前
冷酷的语儿完成签到,获得积分10
10秒前
10秒前
10秒前
10秒前
SIMON发布了新的文献求助10
12秒前
简单海露发布了新的文献求助10
13秒前
LingYue发布了新的文献求助10
13秒前
13秒前
奇奇完成签到,获得积分10
14秒前
和谐沛芹发布了新的文献求助10
14秒前
健壮的尔烟完成签到,获得积分10
14秒前
烟尘完成签到,获得积分10
15秒前
Yv完成签到,获得积分10
16秒前
随便叫什么完成签到,获得积分10
16秒前
17秒前
18秒前
上官若男的应助被娓鸢采纳,获得10
18秒前
19秒前
稳重的无招完成签到,获得积分10
20秒前
活力的念露完成签到,获得积分10
20秒前
20秒前
高分求助中
(应助此贴封号)【重要!!请各用户(尤其是新用户)详细阅读】【科研通的精品贴汇总】 10000
拉伸高分子熔体中缠结行为的模拟和小角中子散射研究 2000
Handbook of Sport Coaching Research 1000
Orthognathic Surgery - 2 Volume Set 500
高分子基功能界面水蒸发材料与应用研究 500
外场增强的水汽发电研究 500
基于结构调控的功能化水诱导发电材料与器件研究 500
热门求助领域 (近24小时)
化学 材料科学 医学 生物 纳米技术 计算机科学 工程类 有机化学 化学工程 生物化学 物理 复合材料 内科学 催化作用 光电子学 人工智能 细胞生物学 无机化学 物理化学 数学
热门帖子
关注 科研通微信公众号,转发送积分 8193361
求助须知:如何正确求助?哪些是违规求助? 9579602
关于积分的说明 21866201
捐赠科研通 7792983
什么是DOI,文献DOI怎么找? 3437263
关于科研通互助平台的介绍 2560718
邀请新用户注册赠送积分活动 2484082