The quality and qualities of classroom observation systems

Teaching quality is often measured by means of classroom observation using a classroom observation system (COS). Generating valid and reliable scores when using a COS is not self-explanatory. Many authors point to issues that need to be taken into account when developing, selecting, and using a COS in order to generate valid and reliable scores. A framework that brings together these issues was developed in the first study of this dissertation. The first part of the framework was designed to evaluate the extent to which the design of a COS will allow it to meet the various criteria set by users. The second and third parts of the framework were designed to be used for assessing the reliability and validity of a COS in a specific context. In the second study of this dissertation, the framework was used to review COSs that were available to measure teaching quality in primary education. An extensive worldwide literature search for COSs, resulting in 27 COSs that met the inclusion criteria. All COSs were reviewed by two reviewers. Reviewers were, on average, positive about the scoring tools, however, insufficient empirical evidence was found for the reliable and valid use of the scores for most COSs. In the third study, the teaching quality scores awarded by external raters, students, and teachers were compared. Little is known about how these three relate, whereas such knowledge may have important implications for measuring teaching quality. Data were collected in 25 grade nine mathematics classrooms. Students' and teachers' perceptions were measured for three lessons per participating teacher, and these lessons were also rated by external raters. The results showed only a significant but low correlation between external raters and students. Thus, teaching quality scores will differ depending on which of the three methods is used. Those who aim to measure teaching quality should therefore carefully select one, or combine multiple methods. The dissertation ends with implications for COS developers, COS users, and governments and other institutions, as well as recommendations for future research.