Accuracy of Large Language Models to Identify Stroke Subtypes Within Unstructured Electronic Health Record Data

BACKGROUND: While International Classification of Diseases, Tenth Revision codes suffice for identifying stroke events in surveillance, accurately classifying stroke types and subtypes using electronic health records remains challenging due to limitations in structured data. This often necessitates manual review of clinical documentation. This study evaluated whether a large language model, GPT-4o, can accurately identify stroke types and subtypes from unstructured clinical notes. METHODS: We implemented a retrieval-augmented generation framework with GPT-4o to classify stroke types (ischemic versus hemorrhagic) and ischemic stroke subtypes using electronic health records data. The American Heart Association Get With The Guidelines–Stroke registry served as the gold standard. Model development used a 20% subset of Get With The Guidelines–Stroke–linked data from UT Southwestern Medical Center, with the remaining 80% reserved for testing. External validation used data from the Parkland Health and Hospital System. A total of 4123 stroke hospitalizations from January 2019 to August 2023 were included (UT Southwestern Medical Center: n=2047; Parkland Health and Hospital System: n=2076). Three prompting strategies—zero-shot Chain-of-Thought, expert-guided, and instruction-based—were evaluated. Predictions of GPT-4os were compared with classifications made by trained abstractors contributing to the Get With The Guidelines–Stroke registry. RESULTS: In the external validation set, 79.6% of patients had ischemic stroke and 20.4% hemorrhagic. GPT-4o achieved 98% accuracy (95% CI, 0.97–0.99) in classifying stroke type, where accuracy reflects the overall proportion of correctly classified patients. Sensitivity was 0.98 (95% CI, 0.97–0.99), and specificity was 0.97 (95% CI, 0.96–0.98). For ischemic stroke subtypes, sensitivity ranged from 0.40 (95% CI, 0.31–0.49) for cryptogenic to 0.95 (95% CI, 0.93–0.97) for small-vessel occlusion. Specificity ranged from 0.94 (95% CI, 0.92–0.96) for large-artery atherosclerosis to 0.98 (95% CI, 0.97–0.99) for cardioembolism. Zero-shot Chain-of-Thought prompting—requiring minimal human input—performed comparably to more labor-intensive strategies. Consistency analysis revealed > 99% agreement across repeated queries. CONCLUSIONS: GPT-4o demonstrated strong accuracy in classifying stroke types but faced challenges with ischemic subtypes.