Clinical Prediction Models: A Comparison Between Cross-Validation and Bootstrap Approaches for Internal Validation

Clinical prediction models are important tools to support medical decision making. Statistical evaluation through internal validation is an essential step before such a model can be implemented in practice. However, with many internal validation strategies available, in particular various bootstrap and cross-validation variants, it is currently unclear which approach achieves the best out-of-sample performance under varying circumstances. In this thesis three extensive simulation studies are performed, evaluating the impact of outcome prevalence, strength of predictors, and type of modeling algorithm (both regression and tree-based). The ability of the approaches to estimate out-of-sample performance is evaluated in terms of Area under ROC curve, calibration slope and intercept ,Mean Absolute Prediction Error, Root Mean Squared Prediction Error, R2Cox-Snell, R2Tjur and Estimated Calibration Index. The results illustrate that all internal validation approaches show more variability, compared to large sample external validation, for the AUC and R2 measures, but less for the calibration intercept. Moreover, no single internal validation approach outperforms all others across all investigated scenarios and performance measures. This indicates that the optimal strategy is dependent on the specific setting and evaluation metric used. The results show that Harrell’s bootstrap works generally well when using regression-based models. However ,when using tree-based models, which showed a high amount of overfitting, Harrell’s, .632 and .632+ bootstrap display large overestimation, while 5, 10 and 10x10 fold cross-validation approach the external validation estimates.