Evaluating Knowledge Structure-based Adaptive Testing Algorithms and System Development

In recent years, many computerized test systems have been developed for diagnosing students’ learning profiles. Nevertheless, it remains a challenging issue to find an adaptive testing algorithm to both shorten testing time and precisely diagnose the knowledge status of students. In order to find a suitable algorithm, four adaptive testing algorithms, based on ordering theory, item relational structure theory, Diagnosys, and domain experts, were evaluated based on the training sample size, prediction accuracy, and the use of test items by the simulation study with paper-based test data. Based on the results of simulation study, ordering theory has the best performance. An ordering-theory-based knowledge-structure-adaptive testing system was developed and evaluated. The results of this system showed that the two different interfaces, paper-based and computer-based, did not affect the examinees’ performance. In addition, the effect of correct guessing was discussed, and two methods with adaptive testing algorithms were proposed to mitigate this effect. The experimental results showed that the proposed methods improve the effect of correct guessing.