Detecting anomalous behaviors of air traffic controllers in time series of facial expressions and head poses

The increasing traffic in the national airspace could overwhelm Air Traffic Controllers (ATCs) and compromise the airport and airside safety. Time series of facial expressions and head pose of ATCs could capture the temporal patterns of ATCs’ mental and physical states, such as emotions, gaze focus (visual attention), fatigue, confusion, and overwhelming mental workload. Anomalous temporal patterns of ATCs’ states could be predecessors or indicators of dangerous mistakes or improper collaborations between ATCs and pilots. Previous research used subjective measures (Task Load Index, TLX) and physiological measures (EMG, blood pressure monitors) to collect data to analyze ATCs’ mental conditions and human performance. However, these methods can be intrusive and cannot achieve real-time monitoring for practical applications. Compared with biometric sensors such as electromyography (EMG), automatic recognition of facial expression and head pose using computer vision is less intrusive for real-time ATC behavior monitoring. The goal of this paper is to investigate two research questions: 1) how real-time computer vision methods could help recover time series of facial expressions, head poses and eye blinks of ATCs? 2) how time series analysis methods could help identify anomalous behaviors of ATCs for guiding targeted management of ATC teams during busy air traffic operations? The research team conducted over ten simulation experiments in a Terminal Radar Approach Control (TRACON) simulator. In these experiments, twelve experienced (retired) ATCs acted as pseudo-ATCs, and graduate students majoring aviation traffic management at Arizona State University (ASU) acted as pseudo pilots. The research team designed Nominal High Workload and Off-Nominal High Workload scenarios to extract the pattern of controllers’ facial behaviors. The preliminary results showed that a recurrent neural network-based time series analysis method could guide the detection of anomalous behaviors of ATCs and communication patterns between ATC and pilots based on time series extracted by computer vision algorithms from videos.