Flaws in the Development and Validation of a Coronavirus Disease 2019 Prediction Model

TO THE EDITOR—The coronavirus disease 2019 (COVID-19) pandemic has seen the development of a number of clinical prediction models to support assessing disease severity or aiding prognosis. A recent systematic review identified 145 models and concluded that all were at high risk of bias, citing concerns with data quality, statistical analysis, and reporting, leading to the conclusion that none are recommended for use [1]. We therefore read with interest the recent article by Dong and colleagues describing the development of a prediction model for assessing survival in patients with COVID-19 [2]. Unfortunately, we observed a number of concerns in the study that we believe deserve highlighting to readers. In their manuscript, the authors randomly split their data into training and validation sets—a practice widely known to be statistically inefficient [3] as it reduces the sample size for developing the model (increasing the risk of overfitting), while the validation cohort is often too small to evaluate performance. The authors then develop a model on the training cohort containing approximately 75 deaths (60% of 121 deaths) and examining >30 candidate predictors; this will almost certainly lead to an overfit model [4]. The resulting model was then evaluated in the validation cohort that contained only about 46 deaths (40% of 121 deaths), which is much lower than the recommended 100 deaths needed for validation [5]. The preferred approach, making the most of the data, would be to develop the model on all available data, and use methods like bootstrapping or cross-validation to both adjust the performance measures (eg, c-index) for optimism (which are almost certainly too high) and shrink the model coefficients (which will likely be overestimated [6]). Other major analysis concerns include categorization of continuous predictors (which results in loss of information [7]), no mention of missing data, use of lasso followed by “multivariate” [sic] Cox regression to screen predictors for inclusion, incorrect (ie, does not reflect the actual model building process) and confusing implementation of cross-validation on the validation data, weak assessment of model calibration by binning observations, and assessment of both the area under the curve and the c-index (both assess discrimination; however, only the latter is correct for survival outcomes) and no assessment of clinical utility. Our second point relates to model presentation. The authors presented their model as a nomogram presumably to aid clinical uptake [8]. However, for other investigators to externally validate this model in their own data to see, it is important that the model underpinning the nomogram is fully reported, namely all of the regression coefficients and importantly the baseline survival at the timepoints of interest (ie, at 14 and 21 days). This latter information is missing, impeding independent external validation of the model. The TRIPOD Statement (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) for reporting (COVID-19) prediction models should be consulted (www.tripod-statement.org) so that important information is presented when authors describe their study [9]. Accompanying the TRIPOD Statement is an Explanation and Elaboration paper that describes the rationale for reporting, as well as discussing various methodological aspects [10]. Potential conflicts of interest. The authors: No reported conflicts of interest. All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest.