Evaluating data extraction error by a large language model from randomised controlled trials: a large-scale empirical study

数据提取 计算机科学 人工智能 人口 自然语言处理 信息抽取 机器学习 样本量测定 十四行诗 统计 随机对照试验 字错误率 荟萃分析 样品(材料) 数据挖掘 数据收集 不利影响 语言模型 随机效应模型 算法
作者
Shiqi Fan,Ming Chen,Suhail A Doi,Zhangnan Ye,Zhen Peng,Yuan Tian,Chengzhi Zhang,Luis Furuya-Kanamori,Lifeng Lin,Evan Mayo-Wilson,M Hassan Murad,Xiuhong Meng,Chang Xu
出处
期刊:BMJ evidence-based medicine [BMJ]
卷期号:: bmjebm-2025
标识
DOI:10.1136/bmjebm-2025-114044
摘要

OBJECTIVE: To examine the potential errors of a general large language model (LLM) (ie, Claude 3.5 Sonnet) on data extraction from randomised controlled trials (RCTs). DESIGN AND SETTING: An empirical study comparing Claude 3.5 Sonnet extractions against a human-performed verification dataset. The extraction tasks for Claude 3.5 Sonnet were based solely on original RCT portable document format (PDF) files. For PDFs that could not be directly extracted by Claude 3.5 Sonnet, optical character recognition was employed to convert them into text format before extraction. PARTICIPANTS: A random sample of 664 trials was selected from a well-established trial bank and a final data pool was established based on rigorous manual cross-checking as a reference standard. DATA SOURCES: PubMed, EMBASE, Scopus, Web of Science (all databases) and the Cochrane Central Register of Controlled Trials (CENTRAL) up to February 2023. ELIGIBILITY CRITERIA FOR SELECTING STUDIES: RCTs on children involving medication and adverse events. MAIN OUTCOME MEASURES: Claude 3.5 Sonnet was applied to extract the basic information (eg, trial design, population information and source of funding) and adverse outcomes (ie, name of adverse events, number of events). Claude 3.5 Sonnet outputs were compared against the final data pool and all errors were recorded. Results are presented as error rates and with 95% CI, estimated using a generalised linear mixed model. RESULTS: For the 664 trials, a total of 23 069 data cells were extracted via Claude 3.5 Sonnet, with 10 624 for basic information and 12 445 for adverse outcomes. The overall error rate for data extraction was 6.6% (95% CI 5.4% to 8.2%), with 5.7% (95% CI 5.2% to 6.1%) in basic information and 7.6% (95% CI 4.9% to 11.8%) in adverse outcomes. When stratified the 1542 total errors by error types, misallocation (assigning data to incorrect fields; 57.1%, 881/1542) and missed or omitted data (incomplete extraction of available data; 23.2%, 357/1542) accounted for the two most frequent errors, with misallocation occurring more in basic information (53.3%, 470/881), while missed or omitted data occurred more in adverse outcomes (96.1%, 343/357). Post hoc analysis examining the association between trial reporting quality (assessed using Consolidated Standards of Reporting Trials (CONSORT) 2025 and LLM data extraction error rates indicated that higher CONSORT adherence was associated with lower extraction error rates. CONCLUSIONS: The data extraction error of Claude was relatively low, but it alerts LLM applications in evidence synthesis. Detailed checking for LLM outputs should be the primary consideration for evidence synthesisers.
最长约 10秒,即可获得该文献文件

科研通智能强力驱动
Strongly Powered by AbleSci AI
科研通是完全免费的文献互助平台,具备全网最快的应助速度,最高的求助完成率。 对每一个文献求助,科研通都将尽心尽力,给求助人一个满意的交代。
实时播报
刚刚
qingniujushi完成签到,获得积分10
刚刚
在水一方的应助被uKey采纳,获得10
2秒前
田様的应助被saturn采纳,获得10
2秒前
2秒前
3秒前
3秒前
第二支羽毛完成签到,获得积分10
4秒前
自然如松完成签到,获得积分10
4秒前
syvshc的应助被vavel采纳,获得10
4秒前
4秒前
科研通AI65的应助被vavel采纳,获得10
4秒前
GIN的应助被zzk采纳,获得10
5秒前
打打的应助被tangerine采纳,获得10
5秒前
5秒前
5秒前
6秒前
yy完成签到,获得积分10
6秒前
7秒前
lxl完成签到,获得积分10
8秒前
8秒前
dlxxx完成签到,获得积分20
8秒前
风吹麦浪的应助被禾乃采纳,获得10
8秒前
9秒前
9秒前
9秒前
123完成签到,获得积分10
9秒前
南柯发布了新的文献求助10
9秒前
谦让音响发布了新的文献求助10
10秒前
张楠完成签到,获得积分10
10秒前
易易发布了新的文献求助10
10秒前
zhuwei发布了新的文献求助10
11秒前
lifan发布了新的文献求助10
11秒前
娓鸢发布了新的文献求助10
11秒前
云念完成签到,获得积分10
11秒前
12秒前
nike胡咧咧完成签到,获得积分10
13秒前
丘比特的应助被randong采纳,获得10
13秒前
13秒前
saturn发布了新的文献求助10
14秒前
高分求助中
(应助此贴封号)【重要!!请各用户(尤其是新用户)详细阅读】【科研通的精品贴汇总】 10000
拉伸高分子熔体中缠结行为的模拟和小角中子散射研究 2000
Handbook of Sport Coaching Research 1000
Orthognathic Surgery - 2 Volume Set 500
高分子基功能界面水蒸发材料与应用研究 500
外场增强的水汽发电研究 500
基于结构调控的功能化水诱导发电材料与器件研究 500
热门求助领域 (近24小时)
化学 材料科学 医学 生物 纳米技术 计算机科学 工程类 有机化学 化学工程 生物化学 物理 复合材料 内科学 催化作用 光电子学 人工智能 细胞生物学 无机化学 物理化学 数学
热门帖子
关注 科研通微信公众号,转发送积分 8193361
求助须知:如何正确求助?哪些是违规求助? 9579602
关于积分的说明 21866201
捐赠科研通 7792983
什么是DOI,文献DOI怎么找? 3437263
关于科研通互助平台的介绍 2560718
邀请新用户注册赠送积分活动 2484082