Development and Evaluation of a Model to Manage Patient Portal Messages

BACKGROUND: Although patient portal messaging in the electronic health record (EHR) can provide convenience and enhanced patient-provider communication, the rising volume of messages in recent years poses challenges, including delays in timely communication and the potential for provider burnout. We evaluated the impact of a natural language processing (NLP) algorithm for automated message routing in clinical practice. METHODS: We developed an NLP model to label and route patient-sent messages using a pretrained classifier that was fine-tuned using clinician feedback. The model was prospectively deployed in an outpatient clinic environment for real-world validation. A parallel group of unrouted messages was retrieved for comparison. The primary end points were time-to-first-message interaction, time-to-conversation resolution, and the total number of message interactions by health care staff, compared with those of the control group. Secondary end points were the precision, recall, F1 score (a measure of positive predictive value and sensitivity), and accuracy for correct message classification. RESULTS: The model prospectively labeled and routed 469 unique conversations over 14 days. Compared with a control group of 402 unrouted conversations from the same period, staff in the routed message group used less time to initially address a new patient message (difference in medians, -1 hour; 95% confidence interval [CI], -1.42 to -0.5) and to complete a conversation (difference in medians, -22.5 hours; 95% CI, -36.3 to -17.7); routed group staff also had fewer total message interactions (difference in medians, -2.0 interactions; 95% CI, -2.9 to -1.4). The model demonstrated high precision (≥96.9%), recall (≥95.0%), and F1 scores (≥96.5%) for accurate prediction of all five message classes, with a total accuracy of 97.8%. CONCLUSIONS: Real-time message routing using an NLP model was associated with reduced message response and resolution times and fewer overall message interactions among clinic staff. (Funded by Switchboard, MD.).