Multimodal machine learning for construction accident severity prediction: An interpretable ensemble framework

Severe construction accidents continue to impose substantial human and economic costs, yet most existing severity prediction models rely primarily on structured variables and overlook the contextual information embedded in narrative reports. This study addresses this limitation by proposing an explainable multimodal machine learning framework that integrates structured injury attributes with enriched narrative-based features to predict post-accident hospitalization outcomes. The framework was developed using 17,270 construction-related incident reports submitted to the U.S. Occupational Safety and Health Administration (OSHA) between 2015 and 2024. Cases were classified as hospitalized or non-hospitalized, where the latter outcomes that did not involve hospital admission. Accident narratives were processed through a text mining and sentiment analysis pipeline, where sentiment reflects the severity and intensity of descriptive language rather than emotional tone. These text-derived features were combined with structured variables to form a multimodal feature space. Class imbalance was addressed using SMOTETomek, and model hyperparameters were optimized through Bayesian Optimization. Three ensemble classifiers—Random Forest, Gradient Boosting Machine, and eXtreme Gradient Boosting—were evaluated against a logistic regression baseline. Among the models tested, Random Forest achieved the strongest performance, with a weighted F1-score of 0.9738 and an AUC of 0.9769. Model interpretability was ensured using SHapley Additive exPlanations (SHAP), which linked predictions to key injury-related factors, including Nature of Injury, Part of Body, Event Type, and narrative sentiment. Although the framework is trained on post-incident data, the extracted patterns and interpretable feature relationships enable proactive risk reduction by identifying recurring high-risk mechanisms, work conditions, and body regions that can be targeted through preventive controls, training, and safety planning. This study extends prior OSHA-based multimodal research by systematically refining narrative features and embedding explainability to support transparent, data-driven construction safety management.