Performance of Machine Learning Models for Sepsis and Stroke Detection Using EMS Data

Early sepsis and stroke recognition by emergency medical services (EMS) improves triage, treatment, and patient outcomes. Machine learning (ML) may enhance EMS sepsis and stroke screening by connecting complex patterns in routinely collected EMS electronic health record (EHR) data, yet research on the application of ML to EMS remains limited. This study evaluated the feasibility and performance of ML sepsis and stroke detection models designed for hospital emergency department (ED) triage data when applied to EMS EHR data. This retrospective analysis included adult ED patients transported by EMS to a single hospital between July 1, 2021, and June 30, 2024, with linked EMS and ED records. Hospital ML models were applied to the EMS EHR data through feature mapping, including vital signs, chief complaints and EMS impressions. ED physician diagnosis served as the reference standard. Given serial EMS vital sign entries for each patient, the ML models could make multiple predictions for each patient: One approach to analysis required a majority of predictions to be indicative of sepsis or stroke; another approach allowed any single prediction to be indicative. Additional analyses allowed the ML models to incorporate the EMS free-text narratives. Model performance was assessed using traditional diagnostic measures (sensitivity, specificity, likelihood ratios (LRs)) and common ML metrics (accuracy, F1 score). Among 27,936 included EMS transports, there were 1,497 (5.4%) sepsis and 729 (2.6%) stroke diagnoses. The primary analysis (“majority of predictions”) model for sepsis demonstrated 68% sensitivity and 86% specificity (+LR: 5, -LR: 0.4; accuracy: 85%, F1: 0.33). The primary analysis stroke model demonstrated 71% sensitivity and 94% specificity (+LR: 11, -LR: 0.3; accuracy: 93%, F1: 0.34). The secondary analysis (“any prediction”) models increased sensitivity (sepsis 75%, stroke 72%) but decreased specificity (sepsis 80%; stroke 93%). Incorporating free-text narratives further increased sensitivity with corresponding decreases in specificity. This study demonstrates the feasibility of applying machine learning models trained on ED triage data to prehospital EMS data for the early identification of stroke and sepsis. Future work should focus on model retraining using EMS-specific data, prospective validation, and real-world implementation strategies.