Gauging Q-Matrix Design and Model Selection in Applied Cognitive Diagnosis

Educational testing has been criticized for its disconnect from modern cognitive science and its limited role in improving instruction and student learning. Reform efforts emphasize the need for testing to provide specific diagnostic insights into students' skills and knowledge. Cognitive diagnosis (CD), an emerging paradigm in educational measurement, addresses these concerns by focusing on instructional content and offering immediate feedback on students' strengths and areas for improvement. CD conceptualizes ability as a collection of discrete latent (cognitive) skills that an examinee may or may not have mastered. Most CD applications in educational testing are confirmatory in nature, requiring the latent skill structure defining ability to be specified a priori, which demands extensive familiarity with the knowledge domain—expertise that may not always be available. This study proposes a new framework leveraging recent methods developed to identify and refine latent skill structures in CD.