Recognising and explaining mental workload using low-interference method by fusing speech, ECG and eye tracking signals during simulated flight

The mental workload of pilots is a critical factor influencing performance. This study aims to propose a low-interference mental workload recognition method based on aviation ecological validity. A simulated flight tracking experiment was conducted with twenty-six pilot cadets. Speech, ECG and eye-tracking data were collected at different workload states corresponding to various training stages. The recognition performance of different feature combinations was evaluated using various machine learning model. Furthermore, the SHapley Additive exPlanations (SHAP) method was used to analyse the relationship between the features and workload state. The results indicate that MFCC_13, Mean NN, W and average pupil diameter were the most influential features in the recognition model. The highest recognition accuracy of 87.36% was achieved using the random forest method trained with full modalities. This study addresses the gap in the field of mental workload recognition regarding low-interference recognition methods and demonstrates its feasibility.