Evaluating the Efficacy of ChatGPT in Navigating the Spanish Medical Residency Entrance Examination (MIR): Promising Horizons for AI in Clinical Medicine

逻辑回归 医学 医疗保健 家庭医学 医学教育 人工智能 内科学 计算机科学 经济增长 经济
作者
Francisco Guillén-Grima,Sara Guillén-Aguinaga,Laura Guillen-Aguinaga,Rosa Alas-Brun,Luc Onambele,Wilfrido Ortega,Rocío Montejo,Enrique Aguinaga-Ontoso,Paul Barach,Inés Aguinaga-Ontoso
出处
期刊:Clinics and practice [MDPI AG]
卷期号:13 (6): 1460-1487 被引量:4
标识
DOI:10.3390/clinpract13060130
摘要

The rapid progress in artificial intelligence, machine learning, and natural language processing has led to increasingly sophisticated large language models (LLMs) for use in healthcare. This study assesses the performance of two LLMs, the GPT-3.5 and GPT-4 models, in passing the MIR medical examination for access to medical specialist training in Spain. Our objectives included gauging the model’s overall performance, analyzing discrepancies across different medical specialties, discerning between theoretical and practical questions, estimating error proportions, and assessing the hypothetical severity of errors committed by a physician. Material and methods: We studied the 2022 Spanish MIR examination results after excluding those questions requiring image evaluations or having acknowledged errors. The remaining 182 questions were presented to the LLM GPT-4 and GPT-3.5 in Spanish and English. Logistic regression models analyzed the relationships between question length, sequence, and performance. We also analyzed the 23 questions with images, using GPT-4’s new image analysis capability. Results: GPT-4 outperformed GPT-3.5, scoring 86.81% in Spanish (p < 0.001). English translations had a slightly enhanced performance. GPT-4 scored 26.1% of the questions with images in English. The results were worse when the questions were in Spanish, 13.0%, although the differences were not statistically significant (p = 0.250). Among medical specialties, GPT-4 achieved a 100% correct response rate in several areas, and the Pharmacology, Critical Care, and Infectious Diseases specialties showed lower performance. The error analysis revealed that while a 13.2% error rate existed, the gravest categories, such as “error requiring intervention to sustain life” and “error resulting in death”, had a 0% rate. Conclusions: GPT-4 performs robustly on the Spanish MIR examination, with varying capabilities to discriminate knowledge across specialties. While the model’s high success rate is commendable, understanding the error severity is critical, especially when considering AI’s potential role in real-world medical practice and its implications for patient safety.
最长约 10秒,即可获得该文献文件

科研通智能强力驱动
Strongly Powered by AbleSci AI
更新
大幅提高文件上传限制,最高150M (2024-4-1)

科研通是完全免费的文献互助平台,具备全网最快的应助速度,最高的求助完成率。 对每一个文献求助,科研通都将尽心尽力,给求助人一个满意的交代。
实时播报
着急的枫叶应助Cloudyyy采纳,获得10
刚刚
完美世界应助科研通管家采纳,获得10
2秒前
充电宝应助tingalan采纳,获得10
2秒前
zy大章鱼完成签到,获得积分10
3秒前
5秒前
指尖心事完成签到 ,获得积分10
6秒前
饱满的灰狼完成签到,获得积分10
8秒前
GG完成签到 ,获得积分10
9秒前
jiangmingjiao发布了新的文献求助10
10秒前
maxyer完成签到,获得积分10
11秒前
wei1390完成签到,获得积分10
12秒前
猫毛完成签到,获得积分10
13秒前
hu完成签到,获得积分10
14秒前
sia完成签到 ,获得积分10
15秒前
蕉鲁诺蕉巴纳完成签到,获得积分10
19秒前
马騳骉完成签到,获得积分10
23秒前
Logan完成签到,获得积分10
26秒前
平淡的寄风完成签到,获得积分10
26秒前
Iris完成签到,获得积分10
27秒前
小徐的日常完成签到 ,获得积分10
30秒前
小鹿儿完成签到,获得积分10
30秒前
论文中完成签到 ,获得积分10
33秒前
韭菜盒子完成签到,获得积分20
33秒前
sjsuA完成签到,获得积分10
33秒前
LCC完成签到 ,获得积分10
33秒前
小城故事和冰雨完成签到,获得积分10
34秒前
秦文静完成签到,获得积分10
36秒前
BaooooooMao完成签到,获得积分10
36秒前
袁国锋完成签到 ,获得积分10
39秒前
雯雯完成签到,获得积分10
40秒前
我我我我我完成签到,获得积分20
44秒前
YY完成签到,获得积分10
45秒前
11号楼203完成签到,获得积分10
55秒前
Jundy完成签到,获得积分10
55秒前
lxhhh完成签到,获得积分10
1分钟前
窗外飞仙发布了新的文献求助10
1分钟前
魔幻千秋完成签到,获得积分0
1分钟前
doyle完成签到,获得积分10
1分钟前
科研小郭完成签到,获得积分10
1分钟前
WTX完成签到,获得积分0
1分钟前
高分求助中
Thermodynamic data for steelmaking 3000
Teaching Social and Emotional Learning in Physical Education 900
藍からはじまる蛍光性トリプタンスリン研究 400
Cardiology: Board and Certification Review 400
[Lambert-Eaton syndrome without calcium channel autoantibodies] 340
NEW VALUES OF SOLUBILITY PARAMETERS FROM VAPOR PRESSURE DATA 300
Transformerboard III 300
热门求助领域 (近24小时)
化学 材料科学 医学 生物 有机化学 工程类 生物化学 纳米技术 物理 内科学 计算机科学 化学工程 复合材料 遗传学 基因 物理化学 催化作用 电极 光电子学 量子力学
热门帖子
关注 科研通微信公众号,转发送积分 2361907
求助须知:如何正确求助?哪些是违规求助? 2069816
关于积分的说明 5170051
捐赠科研通 1798003
什么是DOI,文献DOI怎么找? 897964
版权声明 557715
科研通“疑难数据库(出版商)”最低求助积分说明 479304