Evaluation of the Performance of Survival Analysis Models: Discrimination and Calibration Measures

Prediction functions or Health Risk Appraisal Functions (HRAF) are mathematical models of the probability of an event. This chapter focuses on the evaluation of the performance of an HRAF with regard to its ability to predict the outcome variable. It considers a time to event survival model with censored observations, such as the Cox regression model. The performance of a model with regard to its discrimination and calibration is evaluated. Discrimination refers to a model's ability to correctly distinguish the two classes of outcomes. A model with good discrimination ability produces higher predicted probabilities to subjects who had events than subjects who did not have events. Perfect discrimination would result in two non-overlapping sets of predicted probabilities from the model: one set for the positive outcomes and the other for the negative outcomes. Calibration describes how closely the predicted probabilities agree numerically with the actual outcomes. A model is well calibrated when the predicted and observed values agree for any reasonable grouping of the observation, ordered by increasing predicted values. Calibration measures are often statistics that partition a data set into groups and assess how the average predicted probability compares with the outcome prevalence in each group.