Effect of uncertainty-aware artificial intelligence models on pharmacists’ reaction time and decision-making in a web-based mock medication verification task: A randomized controlled trial (Preprint)

任务(项目管理) 计算机科学 预印本 随机对照试验 眼镜蛇 医学 万维网 工程类 外科 系统工程 程序设计语言
作者
Corey A. Lester,Brigid Rowell,Yifan Zheng,Zoe Co,Vincent D. Marshall,Jin Yong Kim,Qiyuan Chen,Raed Al Kontar,X. Jessie Yang
出处
期刊:JMIR medical informatics [JMIR Publications]
卷期号:13: e64902-e64902
标识
DOI:10.2196/64902
摘要

Abstract Background Artificial intelligence (AI)–based clinical decision support systems are increasingly used in health care. Uncertainty-aware AI presents the model’s confidence in its decision alongside its prediction, whereas black-box AI only provides a prediction. Little is known about how this type of AI affects health care providers’ work performance and reaction time. Objective This study aimed to determine the effects of black-box and uncertainty-aware AI advice on pharmacist decision-making and reaction time. Methods Recruitment emails were sent to pharmacists through professional listservs describing a web-based, crossover, randomized controlled trial. Participants were randomized to the black-box AI or uncertainty-aware AI condition in a 1:1 manner. Participants completed 100 mock verification tasks with AI help and 100 without AI help. The order of no help and AI help was randomized. Participants were exposed to correct and incorrect prescription fills, where the correct decision was to “accept” or “reject,” respectively. AI help provided correct (79%) or incorrect (21%) advice. Reaction times, participant decisions, AI advice, and AI help type were recorded for each verification. Likelihood ratio tests compared means across the three categories of AI type for each level of AI correctness. Results A total of 30 participants provided complete datasets. An equal number of participants were in each AI condition. Participants’ decision-making performance and reaction times differed across the 3 conditions. Accurate AI recommendations resulted in the rejection of the incorrect drug 96.1% and 91.8% of the time for uncertainty-aware AI and black-box AI respectively, compared with 81.2% without AI help. Correctly dispensed medications were accepted at rates of 99.2% with black-box help, 94.1% with uncertainty-aware AI help, and 94.6% without AI help. Uncertainty-aware AI protected against bad AI advice to approve an incorrectly filled medication compared with black-box AI (83.3% vs 76.7%). When the AI recommended rejecting a correctly filled medication, pharmacists without AI help had a higher rate of correctly accepting the medication (94.6%) compared with uncertainty-aware AI help (86.2%) and black-box AI help (81.2%). Uncertainty-aware AI resulted in shorter reaction times than black-box AI and no AI help except in the scenario where “AI rejects the correct drug.” Black-box AI did not lead to reduced reaction times compared with pharmacists acting alone. Conclusions Pharmacists’ performance and reaction times varied by AI type and AI accuracy. Overall, uncertainty-aware AI resulted in faster decision-making and acted as a safeguard against bad AI advice to approve a misfilled medication. Conversely, black-box AI had the longest reaction times, and user performance degraded in the presence of bad AI advice. However, uncertainty-aware AI could result in unnecessary double-checks, but it is preferred over false negative advice, where patients receive the wrong medication. These results highlight the importance of well-designed AI that addresses users’ needs, enhances performance, and avoids overreliance on AI.

科研通智能强力驱动
Strongly Powered by AbleSci AI
更新
PDF的下载单位、IP信息已删除 (2025-6-4)

科研通是完全免费的文献互助平台,具备全网最快的应助速度,最高的求助完成率。 对每一个文献求助,科研通都将尽心尽力,给求助人一个满意的交代。
实时播报
今后应助zhanwenlin采纳,获得10
1秒前
1秒前
freedom313514发布了新的文献求助20
2秒前
wllom完成签到,获得积分20
2秒前
3秒前
MANGMANG发布了新的文献求助30
4秒前
含糊的小松鼠完成签到,获得积分10
4秒前
善学以致用应助fukesi采纳,获得10
4秒前
5秒前
5秒前
5秒前
hm777发布了新的文献求助10
5秒前
传奇3应助帅的一批采纳,获得10
6秒前
6秒前
胡子木发布了新的文献求助10
6秒前
wooooo完成签到,获得积分10
6秒前
7秒前
wllom发布了新的文献求助10
7秒前
LONG发布了新的文献求助10
8秒前
8秒前
昏睡的咖啡完成签到,获得积分10
9秒前
10秒前
10秒前
义气代梅发布了新的文献求助10
10秒前
xx发布了新的文献求助10
10秒前
11秒前
贾琪发布了新的文献求助10
12秒前
14秒前
16秒前
16秒前
16秒前
义气代梅完成签到,获得积分10
17秒前
peanut发布了新的文献求助10
17秒前
勤劳的小牛蛙完成签到 ,获得积分10
17秒前
19秒前
zssl完成签到,获得积分10
19秒前
安年发布了新的文献求助10
20秒前
有知发布了新的文献求助10
20秒前
20秒前
20秒前
高分求助中
The Mother of All Tableaux Order, Equivalence, and Geometry in the Large-scale Structure of Optimality Theory 2400
Ophthalmic Equipment Market by Devices(surgical: vitreorentinal,IOLs,OVDs,contact lens,RGP lens,backflush,diagnostic&monitoring:OCT,actorefractor,keratometer,tonometer,ophthalmoscpe,OVD), End User,Buying Criteria-Global Forecast to2029 2000
Optimal Transport: A Comprehensive Introduction to Modeling, Analysis, Simulation, Applications 800
Official Methods of Analysis of AOAC INTERNATIONAL 600
ACSM’s Guidelines for Exercise Testing and Prescription, 12th edition 588
T/CIET 1202-2025 可吸收再生氧化纤维素止血材料 500
Comparison of adverse drug reactions of heparin and its derivates in the European Economic Area based on data from EudraVigilance between 2017 and 2021 500
热门求助领域 (近24小时)
化学 材料科学 医学 生物 工程类 有机化学 生物化学 物理 内科学 纳米技术 计算机科学 化学工程 复合材料 遗传学 基因 物理化学 催化作用 冶金 细胞生物学 免疫学
热门帖子
关注 科研通微信公众号,转发送积分 3952038
求助须知:如何正确求助?哪些是违规求助? 3497457
关于积分的说明 11087593
捐赠科研通 3228096
什么是DOI,文献DOI怎么找? 1784669
邀请新用户注册赠送积分活动 868839
科研通“疑难数据库(出版商)”最低求助积分说明 801198