Clinician Suicide Risk Assessment for Prediction of Suicide Attempt in a Large Health Care System

Importance: Clinical practice guidelines recommend suicide risk screening and assessment across behavioral health settings. The predictive accuracy of real-world clinician assessments for stratifying patients by risk of future suicidal behavior, however, remains understudied. Objective: To evaluate routine clinical suicide risk assessment for prospectively predicting suicide attempt. Design, Setting, and Participants: This electronic health record-based, prognostic study included 89 957 patients (≥5 years of age) with a structured suicide risk assessment (based on the Suicide Assessment Five-step Evaluation and Triage framework) that was documented by 2577 clinicians during outpatient, inpatient, and emergency department encounters at 12 hospitals in the Mass General Brigham health system between July 2019 and February 2023. Main Outcomes and Measures: The primary outcome was an emergency department visit with a suicide attempt code recorded in the electronic health record within 90 days or 180 days of the index suicide risk assessment. The predictive performance of suicide risk assessments was evaluated on a temporal test set first using stratified prevalence (clinicians' overall risk estimates from a single suicide risk assessment item indicating minimal, low, moderate, or high risk) and then using machine learning models (incorporating all suicide risk assessment items). Results: Of the 812 114 analyzed suicide risk assessments from the electronic health record, 58.81% were with female patients and 3.27% were with patients who were Asian, 5.26% were Black, 3.02% were Hispanic, 77.44% were White, and 11.00% were of Other or Unknown race. After suicide risk assessments were conducted during outpatient encounters, the suicide attempt rate was 0.12% within 90 days and 0.22% within 180 days; for inpatient encounters, the rate was 0.79% within 90 days and 1.29% within 180 days; and for emergency department encounters, the rate was 2.40% within 90 days and 3.70% within 180 days. Among patients evaluated during outpatient encounters, clinicians' overall single-item risk estimates had an area under the curve (AUC) value of 0.77 (95% CI, 0.72-0.81) for 90-day suicide attempt prediction; among patients evaluated during inpatient encounters, the AUC was 0.64 (95% CI, 0.59-0.69); and among patients evaluated during emergency department encounters, the AUC was 0.60 (95% CI, 0.55-0.64). Incorporating all clinician-documented suicide risk assessment items (87 predictors) via machine learning significantly increased the AUC for 90-day risk prediction to 0.87 (95% CI, 0.83-0.90) among patients evaluated during outpatient encounters, 0.79 (95% CI, 0.74-0.84) among patients evaluated during inpatient encounters, and 0.76 (95% CI, 0.72-0.80) among patients evaluated during emergency department encounters. Performance was similar for 180-day suicide risk prediction. The positive predictive values for the best-performing machine learning models (with 95% specificity) ranged from 3.6 to 10.1 times the prevalence for suicide attempt. Conclusions and Relevance: Clinicians stratify patients for suicide risk at levels significantly above chance. However, the predictive accuracy improves significantly by statistically incorporating information about recent suicidal thoughts and behaviors and other factors routinely assessed during clinical suicide risk assessment.