Reliability of observers' subjective impressions of families: A generalizability theory approach

Parenting was observed in videotaped interactions in 30 families referred for child conduct problems. Generalizability coefficients and the impact of varying numbers of raters were estimated. Two measurement designs were compared: All raters observed all families ("crossed" design) and a different rater observed each family ("nested" design). The crossed design provided higher generalizability coefficients than a nested design, implying inflated generalizability estimates if a crossed estimation model is used for a nested data collection. Three and four raters were needed to obtain generalizability coefficients in the .70-.80 range for monitoring and discipline, respectively. One rater was sufficient for a corresponding estimate for positive involvement and for an estimate in .80-.90 range for problem-solving. Estimates for skill encouragement were non-acceptable.