Assessing Intervention Timing in Computer-Based Education Using Machine Learning Algorithms

The use of computer-based and online education systems has made new data available that can describe the temporal and process-level progression of learning. To date, machine learning research has not considered the impacts of these properties on the machine learning prediction task in educational settings. Machine learning algorithms may have applications in supporting targeted intervention approaches. The goals of this paper are to: 1) determine the impact of process-level information on machine learning prediction results and 2) establish the effect of type of machine learning algorithm used on prediction results. Data were collected from a university level course in human factors engineering (n=35), which included both traditional classroom assessment and computer-based assessment methods. A set of common regression and classification algorithms were applied to the data to predict final course score. The overall prediction accuracy as well as the chronological progression of prediction accuracy was analyzed for each algorithm. Simple machine learning algorithms (linear regression, logistic regression) had comparable performance with more complex methods (support vector machines, artificial neural networks). Process-level information was not useful in post-hoc predictions, but contributed significantly to allowing for accurate predictions to be made earlier in the course. Process level information provides useful prediction features for development of targeted intervention techniques, as it allows more accurate predictions to be made earlier in the course. For small course data sets, the prediction accuracy and simplicity of linear regression and logistic regression make these methods preferable to more complex algorithms.