Real-Time Driver Cognitive Workload Recognition: Attention-Enabled Learning With Multimodal Information Fusion

Driver workload inference is significant for the design of intelligent human–machine cooperative driving schemes since it allows the systems to alert drivers before potentially dangerous maneuvers and achieve a safer control transition. However, pattern variations among individual drivers and sensor artifacts pose great challenges to the existing cognitive workload recognition approaches. In this article, we develop an attention-enabled recognition network with a decision-level fusion architecture to further improve the workload estimation performance. Specifically, the cross-attention mechanism can enhance useful feature representations learned by hyper long-short-term-memory-based modules from time-series multimodal information, i.e., electroencephalogram signals, eye movements, and vehicle states. A novel dataset containing multiple driving scenarios is constructed to evaluate the model performance across different historical horizons and decision thresholds, and test results demonstrate the superior performance of the proposed model to other existing methods. Furthermore, robustness tests and driver-in-the-loop experiments are conducted to verify the effectiveness of the developed model in real-time workload levels inference. The code and supplementary materials are available at https://yanghh.io/Driver-Workload-Recognition .