Measurement and Real-Time Recognition of Driver Trust in Conditionally Automated Vehicles: Using Multimodal Feature Fusions Network

Trust calibration is essential to prevent misuse and disuse of automated vehicles (AVs). Accurate measurement and real-time identification of driver trust is an important prerequisite for achieving trust calibration. Currently, in conditionally automated driving, most researchers utilize self-reported ratings as the ground truth to evaluate driver trust and explore objective trust indicators. However, inconsistencies between the subjective rating and objective behaviors were reported, indicating that trust measurements cannot rely solely on self-reported ratings. To fill this research gap, a method of subjective and objective combination was proposed to measure and identify driver trust in AVs. Thirty-four drivers were involved in a sequence of takeover events. Monitoring ratio and subjective trusting ratings were collected, and combined to measure driver trust levels (i.e., higher and lower trust). Compared with the subjective measurement, the hybrid measurement can more reliably evaluate driver trust in AVs. More importantly, we established a real-time driver trust recognition model for AVs using label smoothing-based convolutional neural network and long short-term memory network fusing multimodal physiological signals (i.e., galvanic skin response and electrocardiogram) and interactive experiences (i.e., takeover-related lead time, takeover frequencies and system usage time). Four common models were developed to compare with the proposed model: Gaussian naive Bayes, support vector machine, convolutional neural network, and long short-term memory network. The comparison results suggest that the performance of our model outperforms others with an F1-score of 75.3% and an area under curve value of 0.812. These findings could have implications for the development of trust monitoring systems in conditionally automated driving.