Vehicle real-time collision risk prediction: A multi-modal learning approach for diverse urban road scenarios based on a large-scale near-crash event dataset

The effectiveness of vehicle collision avoidance systems depends on the precision of collision risk prediction models. However, current models often neglect the driver's condition, resulting in their poor ability to predict near-crash events triggered by aggressive, fatigued, or distracted driving. Additionally, current models overlook the differences in modality, type, and variability of multi-source data, leading to insufficient feature extraction from input data, which in turn limits the model's prediction accuracy. To address these issues, we developed an end-to-end pre-trained deep framework (PM-Transformer) with a Transformer, consisting of multi-module recurrent convolutional neural networks. The framework includes four modules: (1) pre-trained time series module that extracts spatiotemporal information from traffic time series using one-dimensional convolutional neural network - long short-term memory; (2) pre-trained spectral module that learns visual temporal representations from traffic spectrograms using two-dimensional convolutional neural network - long short-term memory; (3) metadata module for vectorizing traffic metadata; (4) fusion module that semantically integrates features from the three modules using a Transformer. Results show that the proposed model can achieve the same prediction accuracy as other models using only 5 % of their training sample size. Compared to other traditional models, our model improves the accuracy of risk prediction by 6 %, 9 %, and 4 %, respectively, with small sample sizes (0.5 s, 1 s, and 2 s in advance), while also maintaining the best performance with larger sample sizes. Findings of this study hold significant potential for improving the effectiveness of vehicle collision avoidance systems. • The study enhances collision risk prediction using driver, vehicle, and road conditions. • The PM-Transformer model achieves high accuracy with only 5% of the training data.