医学
再培训
接收机工作特性
曲线下面积
急诊医学
重症监护
校准
机器学习
预测建模
人工智能
重症监护医学
统计
内科学
计算机科学
国际贸易
业务
数学
作者
Anne de Hond,Ilse Kant,Mattia Fornasa,Giovanni Ciná,Paul Elbers,Patrick Thoral,M. Sesmu Arbous,Ewout W. Steyerberg
标识
DOI:10.1097/ccm.0000000000005758
摘要
Many machine learning (ML) models have been developed for application in the ICU, but few models have been subjected to external validation. The performance of these models in new settings therefore remains unknown. The objective of this study was to assess the performance of an existing decision support tool based on a ML model predicting readmission or death within 7 days after ICU discharge before, during, and after retraining and recalibration.A gradient boosted ML model was developed and validated on electronic health record data from 2004 to 2021. We performed an independent validation of this model on electronic health record data from 2011 to 2019 from a different tertiary care center.Two ICUs in tertiary care centers in The Netherlands.Adult patients who were admitted to the ICU and stayed for longer than 12 hours.None.We assessed discrimination by area under the receiver operating characteristic curve (AUC) and calibration (slope and intercept). We retrained and recalibrated the original model and assessed performance via a temporal validation design. The final retrained model was cross-validated on all data from the new site. Readmission or death within 7 days after ICU discharge occurred in 577 of 10,052 ICU admissions (5.7%) at the new site. External validation revealed moderate discrimination with an AUC of 0.72 (95% CI 0.67-0.76). Retrained models showed improved discrimination with AUC 0.79 (95% CI 0.75-0.82) for the final validation model. Calibration was poor initially and good after recalibration via isotonic regression.In this era of expanding availability of ML models, external validation and retraining are key steps to consider before applying ML models to new settings. Clinicians and decision-makers should take this into account when considering applying new ML models to their local settings.
科研通智能强力驱动
Strongly Powered by AbleSci AI