Classification of carrier-based aircraft pilot mental workloads based on feature-level fusion and decision-level fusion of PPG and EEG signals

Abstract This study explored mental workload recognition methods for carrier-based aircraft pilots utilising multiple sensor physiological signal fusion and portable devices. A simulation carrier-based aircraft flight experiment was designed, and subjective mental workload scores and electroencephalogram (EEG) and photoplethysmogram (PPG) signals from six pilot cadets were collected using NASA Task Load Index (NASA-TLX) and portable devices. The subjective scores of the pilots in three flight phases were used to label the data into three mental workload levels. Features from the physiological signals were extracted, and the interrelations between mental workload and physiological indicators were evaluated. Machine learning and deep learning algorithms were used to classify the pilots’ mental workload. The performances of the single-modal method and multimodal fusion methods were investigated. The results showed that the multimodal fusion methods outperformed the single-modal methods, achieving higher accuracy, precision, recall and F1 score. Among all the classifiers, the random forest classifier with feature-level fusion obtained the best results, with an accuracy of 97.69%, precision of 98.08%, recall of 96.98% and F1 score of 97.44%. The findings of this study demonstrate the effectiveness and feasibility of the proposed method, offering insights into mental workload management and the enhancement of flight safety for carrier-based aircraft pilots.