An imputation method for estimating the learning curve in classification problems

The learning curve expresses the error rate of a predictive modeling procedure as a function of the sample size of the training dataset. It typically is a decreasing, convex function with a positive limiting value. An estimate of the learning curve can be used to assess whether a modeling procedure should be expected to become substantially more accurate if additional training data become available. This article proposes a new procedure for estimating learning curves using imputation. We focus on classification, although the idea is applicable to other predictive modeling settings. Simulation studies indicate that the learning curve can be estimated with useful accuracy for a roughly four-fold increase in the size of the training set relative to the available data, and that the proposed imputation approach outperforms an alternative estimation approach based on parameterizing the learning curve. We illustrate the method with an application that predicts the risk of disease progression for people with chronic lymphocytic leukemia.