Measures of Agreement: Reliability, Classification Accuracy, and Classification Consistency

Gierl, Cui, and Zhou (J Educ Meas 46:293–313, 2009), Cui, Gierl, and Chang (J Educ Meas 49:19–38, 2012), Templin and Bradshaw (J Classif 30:251–275, 2013), Wang, Song, Chen, Meng, and Ding (J Educ Meas 52:457–476, 2015), Johnson and Sinharay (J Educ Meas, 55: 635–664, 2018), and Johnson and Sinharay (J Educ Behav Stat, in press) suggested reliability-like measures for the estimates obtained from a diagnostic classification model. These measures mostly express the agreement between the estimated skill and the true skill, or between estimated skills from parallel assessments. This paper provides a review of these measures and demonstrates some of them for a real data example.