Predicting Performance of First-Year Residents: Correlations Between Structured Interview, Licensure Exam, and Competency Scores in a Multi-Institutional Study

Purpose To determine whether scores on structured interview (SI) questions designed to measure noncognitive competencies in physicians (1) predict subsequent first-year resident performance on Accreditation Council for Graduate Medical Education (ACGME) milestones and (2) add incremental validity over United States Medical Licensing Examination (USMLE) Step 1 and Step 2 Clinical Knowledge scores in predicting performance. Method The authors developed 18 behavioral description questions to measure key noncognitive competencies (e.g., teamwork). In 2013–2015, 14 programs (13 residency, 1 fellowship) from 6 institutions used subsets of these questions in their selection processes. The authors conducted analyses to determine the validity of SI and USMLE scores in predicting first-year resident milestone performance in the ACGME’s core competency domains and overall. Results SI scores predicted midyear and year-end overall performance ( r = 0.18 and 0.19, respectively, P < .05) and year-end performance on patient care, interpersonal and communication skills, and professionalism competencies ( r = 0.23, r = 0.22, and r = 0.20, respectively, P < .05). SI scores contributed incremental validity over USMLE scores in predicting year-end performance on patient care ( ΔR = 0.05), interpersonal and communication skills ( ΔR = 0.09), and professionalism ( ΔR = 0.09; all P < .05). USMLE scores contributed incremental validity over SI scores in predicting year-end performance overall and on patient care and medical knowledge. Conclusions SI scores predict first-year resident year-end performance in the interpersonal and communication skills, patient care, and professionalism competency domains. Future research should investigate whether SIs predict a range of clinically relevant outcomes.