Virtual patient and feedback intervention to improve clinical reasoning for dizziness in the emergency department

Diagnostic errors are universal and a public health problem. Dizziness is a common presentation in the emergency department (ED), costly to assess, and associated with diagnostic errors.1, 2 Knowledge gaps and lack of feedback appear to play an important role,3 suggesting that educational interventions may help mitigate diagnostic errors. Virtual patients (VPs) are computer-based educational programs used to safely practice history-taking, physical examination skills, and diagnostic and therapeutic decision making in simulated clinical scenarios.3 We previously designed an educational intervention using VPs in internal medicine and showed improvement in interns’ ability to diagnose dizziness cases.3 To address the lack of educational and feedback interventions to improve clinical reasoning skills in the ED,4 we piloted a VP-based curriculum with systematic feedback on diagnostic performance for dizziness in the ED. We evaluated it by assessing ED clinicians’ diagnostic accuracy, appropriate neuroimaging utilization, and electronic health record (EHR) documentation of clinical reasoning for patients presenting with dizziness. The study was conducted at two large academic EDs from May 2021 to May 2022. Participation was voluntary, and sample size was based on the availability of subjects. Figure S1 shows the study flow diagram. Phase 1 (May 2021–November 2021) was an unblinded, stratified, randomized controlled, delayed intervention, pretest–posttest study. Participants were stratified based on years of clinical ED experience as junior (≤1 year) or senior (>1 year). They were then randomized 1:1 into group A (immediate intervention) or group B (control/delayed intervention). After taking pretest VP cases, group A received the intervention, and group B was exposed to control. After the intervention, both groups were exposed to posttest VP cases (which differed from pretest cases). Group B was then exposed to the intervention and took posttest cases. Both groups were exposed to a new set of VP cases after 6 months (test of retention). Phase 2 (December 2021–May 2022) was a nonrandomized controlled study comparing all participants who completed Phase 1 (intervention group) versus a clinical experience–matched controlled group of ED clinicians who did not participate in our study (e.g., intervention group resident with 1 year of clinical experience in the ED was matched with control group resident with the same year of experience). We compared outcome measures between the two groups before and after the intervention. The study was approved by our institutional review board. The intervention included: (1) An educational activity with five 90-min online sessions approximately every week for 5 weeks (∼7.5 h total). This timeline was based on prior experience training internal medicine residents on dizziness.3, 5 The first was a lecture, followed by three practice sessions with a library of 14 VP dizzy cases (six were pretest cases, and eight were new VP cases) moderated by clinician educators. The final session focused on physical examination using the aVOR mobile app (https://apps.apple.com/us/app/avor/id497245573) to help with dizziness diagnosis. The aVOR app has been shown to improve medical students’ competence in treating benign paroxysmal positional vertigo.6 All sessions followed a deliberate practice model;7 learners were given a well-defined task and multiple opportunities for practice with systematic feedback to correct any errors (feedback was provided on each section of the history, physical examination, test use/interpretation, and the assessment/plan).7 The VP cases included benign, common causes of dizziness such as benign paroxysmal positional vertigo as well as more dangerous etiologies such as stroke (Methods S1). Video S4 demonstrates the VP software. The VP cases were developed and piloted on internal medicine residents, as detailed in our previous publication.3 (2) Chart review with feedback to participants. From May to November 2021, participants were provided the opportunity to receive feedback (Methods S2) via REDCap surveys on dizzy patients they saw in the ED. The control group was instructed to review online videos and articles on dizziness over the 5-week time period (Methods S3). For Phase 1 data analysis, diagnostic accuracy and appropriate neuroimaging utilization (CT head) were measured by participants’ performance on VP cases (pretest vs. posttest vs. test of retention). These were scored as either correct (matches the expert-adjudicated diagnosis) or incorrect (does not match expert diagnosis). Details about the adjudication process can be found in Methods S4. Total diagnostic accuracy and appropriate CT head utilization were calculated as a percentage for each participant. For Phase 2 analysis, we compared participants’ clinical reasoning documentation in the EHR for dizzy patients seen in the ED with a matched control group. The selection of EHR notes and the creation of a rubric is described in Methods S5. Since there is no criterion standard to assess this documentation, we created a rubric that two board-certified ED physicians (blinded to the identity of the groups) used to score a random set of EHR notes written by the intervention and control groups. Not all items within the rubric were scored for all notes. Given this, we calculated totals as a percentage (total points given by the raters over total possible points based on the rubric) for the four rubric domains (history, physical examination, assessment and plan, and total rubric score). We calculated inter-rater reliability using intraclass correlation coefficients for rubric totals. We then used the means of raters to compare intervention and control groups and when calculating Spearman correlation coefficients between rubric totals and the global items. In Phase 1, 14 ED clinicians participated (seven per group). The median diagnostic accuracy and appropriate CT head scores were better for the intervention group (group A) compared with the control group (group B) on posttest cases. After group B was exposed to the intervention as the delayed intervention arm, its scores improved. Both groups showed retention at 6 months (Table 1). Phase 2 compared data on 14 clinicians who completed Phase 1 (intervention group) with 13 in an intervention-and-testing–naïve, matched control group. For Phase 2, a total of 120 EHR notes (40 each from preintervention, during intervention, and postintervention period) with dizziness as the chief complaint were randomly selected to be evaluated. The two ED physician raters separately scored 119 EHR notes (one note was not analyzed due to missing data). Intraclass correlation coefficient for the total scores obtained on the rubric by the two raters was 0.77 (95% CI 0.68–0.84; internal structure validity evidence). The Spearman's rho for the total scores obtained on the rubric and the raters’ assessment of the participants’ overall clinical performance in evaluating the patients with dizziness was 0.77; the rubric total score and the raters’ scoring of the overall clarity/quality of the notes was 0.77 (relationship to other variables validity evidence). The median scores for some sections of the rubric were higher in the intervention group compared with the control group in the postintervention period, though the differences did not reach statistical significance (Table S1). After participating in our study, ED clinicians demonstrated improved diagnostic accuracy and appropriate CT head utilization, with retention of skills 6 months after the intervention. However, they did not demonstrate statistically significant higher scores on clinical reasoning documentation relative to matched controls, as assessed by the dizziness rubric. Systematic reviews8, 9 comparing simulation/VPs with traditional education showed improvement in clinical reasoning skills for VPs (mixed evidence). However, the studies were done with medical students,8 there was inconsistency between studies, and more than half of them focused on procedural training. Retesting of skills at 3 months did not show positive effects.9 Our study adds to this literature by focusing on specific measures of clinical reasoning and assessing for improvements in diagnostic accuracy, appropriate neuroimaging utilization, long-term retention of skills (at 6 months), and EHR documentation. A systematic review highlighted the need for scalable solutions to enhance bedside diagnosis in the ED, particularly for presentations causing serious harm such as dizziness.2 A clinical policy article noted a scarcity of data on the effectiveness of bedside diagnostic features in accurately identifying strokes in dizzy patients, largely due to insufficient studies involving ED physicians.10 The GRACE-3 guideline strongly recommends using physical examination (HINTS) to help diagnose stroke in dizzy patients and urges training for emergency clinicians.1 Our study contributes to these recommendations by evaluating a scalable solution (VPs) aimed at improving emergency clinicians’ ability to differentiate amongst the causes of dizziness and utilize tests judiciously. Our study has several limitations. First, it was conducted at one health care system, focused on one single presenting chief complaint, and faced recruitment difficulties due to the COVID-19 pandemic, leading to a small sample size. Participation was voluntary, which could have led to self-selection bias. Therefore, the findings may not generalize. Second, while participants’ assessments with VP cases and EHR clinical reasoning documentation for dizzy patients seen in the ED are a useful proxy, performance on these variables may not translate into improved clinical reasoning with actual patients. Third, the sample size of both clinicians and case notes was small and based on availability of test subjects and patients during the study period, rather than an a priori power calculation. Higher scores on the EHR clinical reasoning rubric for dizziness were not statistically significantly different between intervention and control groups, and it remains unknown whether these differences may have achieved statistical significance in a larger sample. Fourth, though we assessed EHR notes specifically written by individual clinicians who participated in our intervention versus a matched control group of clinicians, the documentation by trainees in the EHR may have been influenced by the supervising physicians with whom they were working, potentially impacting our results. Finally, the clinicians could have created EHR notes de novo or used a template to document. The use of a template could have influenced the rubric score. We combined two high-yield interventions—deliberate practice with real-world VP cases and systematic feedback on diagnostic performance obtained via chart reviews. Our intervention was conducted online, making it easily scalable; it could be rapidly implemented in a busy ED as it is time efficient and provides the clinician with feedback and opportunities for practice in a safe learning environment. Future research using larger samples should evaluate whether such interventions can effectively enhance clinical reasoning skills and reduce diagnostic errors, specifically with patients presenting to the emergency department with dizziness. Susrutha Kotwal and Rodney Omron conceived and designed the study. Susrutha Kotwal and Shervin Badihian were involved in data collection. Zheyu Wang and Sean Tackett performed the data analysis and were involved for their statistical expertise. Susrutha Kotwal drafted the manuscript. All authors (Susrutha Kotwal, Shervin Badihian, Zheyu Wang, Sean Tackett, Eric Steinberg, Cory Clugston, Susan Peterson, David E. Newman-Toker, and Rodney Omron) were involved in data interpretation and critical revision of the manuscript for important intellectual content. Susrutha Kotwal, Zheyu Wang, Susan Peterson, David E. Newman-Toker, and Rodney Omron were involved in acquisition of funding. All authors approved the final manuscript. The authors report no conflicts of interest. Data available on request from the authors. File S1. Figure S1. Table S1. File S2. Video Caption: VP software demonstration. Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.