Air Traffic Controller Fatigue Detection Based on Facial and Vocal Features Using Long Short-Term Memory

Air traffic controller fatigue has become a significant concern for flight safety. With the sharp rise in global air traffic, it is imperative to assess controller fatigue, as it directly impacts the safety and efficiency of air traffic control operations. Our study introduces a non-intrusive method to detect fatigue by analyzing the facial and vocal characteristics of air traffic controllers. Initially, we developed fast and accurate schemes for facial feature extraction, which allowed us to measure the “percentage of eyelid closures” and yawn frequency from video recordings. Subsequently, we extracted several vocal features from audio recordings, including average fundamental frequency, short-time average magnitude, short-time zero-crossing rate, harmonic-to-noise ratio, jitter, shimmer, loudness, and Mel-frequency cepstrum coefficient. We then created temporal sequences of these facial and vocal features to feed into a bi-directional long short-term memory gated recurrent unit network. This data, combined with the Stanford Sleepiness Scale, facilitated the identification and precise prediction of controller fatigue levels. Our experimental findings validate the effectiveness of the proposed detection method, which demonstrated a recognition accuracy rate of 95.12% on the test audio and video datasets.