Opening the Black Box: Agreement and Reliability of a Situational Judgment Test Across Multiple Institutions

The primary objective of this study was to examine the levels of agreement and reliability of a situational judgment test (SJT) using a diverse pool of pharmacy practice faculty as subject matter experts. Secondary aims included analyses to build support for test validity and fairness. An SJT containing 18 scenarios and 118 responses assessing empathy, integrity, and teamwork was developed and delivered to pharmacy practice faculty at 5 schools of pharmacy across the United States. Reliability was assessed by examining internal consistency, inter-rater reliability, and split-half reliability. Only responses which attained an inter-rater agreement>0.7 were included in the final version of the SJT. All responses were scored using a near-miss system, allowing higher scores for answers more closely aligned with the key, which was determined by the faculty who completed the SJT. Test fairness was reported using descriptive statistics. Thirty-nine faculty across the 5 participating institutions completed the SJT. The final version of the SJT included 105 responses, achieving an inter-rater agreement of>0.7 (inter-rater reliability of 0.98). Split-half reliability was 0.72. The average score was 85.7%, and no differences in performance were observed based on demographic characteristics. An SJT designed to assess empathy, integrity, and teamwork achieved reasonable levels of reliability among pharmacy practice faculty across the United States, and the results provided initial support for test validity and fairness. These results support a pilot to assess this SJT among students representing multiple institutions.