Multimodal Sensing and Computational Intelligence for Situation Awareness Classification in Autonomous Driving

Maintaining situation awareness (SA) is essential for drivers to deal with the situations that Society of Automotive Engineers (SAE) Level 3 automated vehicle systems are not designed to handle. Although advanced physiological sensors can enable continuous SA assessments, previous single-modality approaches may not be sufficient to capture SA. To address this limitation, the current study demonstrates a multimodal sensing approach for objective SA monitoring. Physiological sensor data from electroencephalogram and eye-tracking were recorded for 30 participants as they performed three secondary tasks during automated driving scenarios that consisted of a pre-takeover (pre-TOR) request segment and a post-TOR segment. The tasks varied in terms of how visual attention was allocated in the pre-TOR segment. In the post-TOR segment, drivers were expected to gather information from the driving environment in preparation for a vehicle-to-driver transition. Participants' ground-truth SA level was measured using the Situation Awareness Global Assessment Techniques (SAGAT) after the post-TOR segment. A total of 23 physiological features were extracted from the post-TOR segment to train computational intelligence models. Results compared the performance of five different classifiers, the ground-truth labeling strategies, and the features included in the model. Overall, the proposed neural network model outperformed other machine learning models and achieved the best classification accuracy (90.6%). A model with 11 features was optimal. In addition, the multi-physiological sensor-model outperformed the single sensing model by comparing prediction performance. Our results suggest that multimodal sensing model can objectively predict SA. The results of this study provide new insight into how physiological features contribute to the SA assessment.