Building a Validity Argument for a New Clinical Reasoning Item Format: A Think-Aloud Interview Study of SHARP Item Response Processes

Abstract Purpose The SHARP (SHort Answer, Rationale Provision) question format assesses aspects of students’ clinical reasoning skills. The format focuses on rationale provision or justification of clinical interpretations and decisions. SHARP items present a patient chart and ask students to enter a brief free-text response to a clinical question. Students then must select information from the chart to support their response. This study evaluates response process validity evidence for the extrapolation inference of a validity argument and use of SHARP item performances to make summative evaluations. Method The authors conducted concurrent think-aloud interviews with fourth-year medical students from Liaison Committee on Medical Education–accredited U.S. medical schools from November 2021 to March 2022. Students were asked to verbalize their thoughts as they answered 12 SHARP items. Interviews were audio-recorded, video-recorded, and transcribed verbatim. The authors used deductive and inductive coding approaches and qualitative techniques to analyze the interviews. Results Eighteen fourth-year medical students from 13 Liaison Committee on Medical Education–accredited U.S. medical schools participated in the think-aloud interviews. Students justified their reasoning using 4 primary strategies: (1) identifying associations, (2) explaining associations, (3) thinking probabilistically, and (4) prioritizing evidence. Identifying associations refers to when students connect patient information with conditions and/or diagnoses; explaining associations refers to when students include portrayals of the biological processes and/or complex clinical relationships underlying identified associations; thinking probabilistically refers to when students rely on their knowledge of risk factors, likelihoods, epidemiology, and/or statistics; and prioritizing evidence refers to when students identified the most supportive pieces of patient information. These strategies partially align with established frameworks for understanding clinical reasoning in real-world settings. Conclusions Findings provide initial validity evidence based on response processes for SHARP item score interpretations and uses. Collection and evaluation of additional validity evidence are needed to bolster the SHARP item validity argument.