Confidence Limits for Intraclass Reliability Coefficients

Abstract A reliability coefficient (R) calculated using sample data may differ considerably in value from the population reliability coefficient. The reporting of confidence limits for the population reliability coefficient assists a person in interpreting the goodness of a sample reliability coefficient. The study was designed to (a) present the procedures for calculating confidence limits for R, (b) present confidence limits for several values of R and several sample sizes, and (c) describe a program in SPSS (1999) that provides confidence limits and hypothesis testing for Rs. The confidence limits for 3 different values of R, 2 different ANOVA models, 2 different sample sizes, 2 different numbers of repeated measures, and 2 different criterion scores were calculated. The confidence limits became narrower as R increased in value, as sample size increased in value, and as the number of repeated measures increased. For a given (a) R, (b) sample size, (c) number of repeated measures, and (d) criterion score, the confidence limits are the same for a 1-way and a 2-way Analysis of Variance model. Furthermore, when all other factors are equal, the confidence limits tend to be narrower for a criterion score that is a single score rather than a mean score. Based on the confidence limits reported, a sample size of at least 50 is needed to have relatively narrow confidence limits.