Prospective and End User Evaluation of an AI-based Clinical Decision Support System for Individualized Antimicrobial Stewardship

Introduction: Antimicrobial stewardship is crucial in combating the global problem of antimicrobial resistance. A key aspect of stewardship is intravenous-to-oral switching (IVOS). Research indicates oral antibiotics are often non-inferior to intravenous treatment. However, individual decisions regarding IVOS are complex and under-researched, impacting healthcare costs, patient comfort, and length of stay (LOS). We developed a machine learning model to predict when IVOS may be appropriate for an individual to support antimicrobial stewardship decision making. This clinical decision support system (CDSS) was evaluated through a prospective dataset and a user study involving simulated and real case vignettes. Methods: Patients receiving intravenous antibiotics for less than eight days followed by oral treatment were identified from electronic health record data in Imperial College Healthcare NHS Trust's Clinical Analytics, Research, and Evaluation (iCARE) environment. Patient demographics, co-morbidities, and vital signs were extracted as features. Disease embeddings and a set transformer were employed for processing co-morbidities, while 253 features were derived from vital signs using the 'CAnonical Time-series CHaracteristics' framework. Models were trained and tested using 10-fold cross-validation. A prospective dataset was generated from new data released into iCARE during research (June 2023). The best-performing model from cross-validation was applied to this dataset. Simulated case vignettes, based on iCARE data and real patient cases from MIMIC-IV, were integrated into a web app developed using Django. Clinicians involved in antimicrobial prescribing were recruited to review these cases and provide IVOS decisions. Results: 5,610 unique stays were used for cross-validation with the prospective dataset containing 547. The model achieved an AUROC of 0.77, accuracy of 0.70, true positive rate of 0.61, and false positive rate of 0.21 on the unseen prospective dataset, predicting that 51% of patients could have switched earlier than they did. Minimal differences between the real and predicted switch events correlated with reduced mean patient LOS. A statistically significant (p-value <0.01) difference in remaining LOS was observed between patients receiving oral versus intravenous treatment until five days of intravenous treatment. Data collection for the case vignette study is ongoing and will be completed before ICID. Preliminary qualitative results indicate that if a user trusts the CDSS, it sees increased engagement during decision-making, and can enable faster prescribing choices. Discussion: This CDSS aims to facilitate early switching when appropriate. Our model's predictions align with current guidelines, suggesting that switching in the first 48 hours is generally inappropriate, but subsequent early IVOS can be determined based on a patient's clinical status. Through case vignettes, we aim to quantify differences in prescribing behaviors between the CDSS and standard care, as well as evaluate user acceptability of the technology. Conclusion: AI-based CDSSs have the potential to support antimicrobial stewardship decision-making, though their impact on prescribing and safety remains unanswered.