Interrater reliability: the kappa statistic

Many situations in the healthcare industry rely on multiple people to collect research or clinical laboratory data .The question of consistency, or agreement among the individuals collecting data immediately arises due to the variability among human observers .Well-designed research studies must therefore include procedures that measure agreement among the various data collectors .Study designs typically involve training the data collectors, and measuring the extent to which they record the same scores for the same phenomena .Perfect agreement is seldom achieved, and confidence in study results is partly a function of the amount of disagreement, or error introduced into the study from inconsistency among the data collectors .The extent of agreement among data collectors is called, "interrater reliability" .Interrater reliability is a concern to one degree or another in most large studies due to the fact that multiple people collecting data may experience and interpret the phenomena of interest differently .