Automatic depression recognition using CNN with attention mechanism from videos

Artificial intelligence (AI) has incorporated various automatic systems and frameworks to diagnose the severity of depression using hand-crafted features. However, process of feature selection needs domain knowledge and is still time-consuming and subjective. Deep learning technology has been successfully adopted for depression recognition. Most previous works pre-train the deep models on large databases followed by fine-tuning with depression databases (i.e., AVEC2013, AVEC2014). In the present paper we propose an integrated framework – Deep Local Global Attention Convolutional Neural Network (DLGA-CNN) for depression recognition, which adopts CNN with attention mechanism as well as weighted spatial pyramid pooling (WSPP) to learn a deep and global representation. Two branches are introduced: Local Attention based CNN (LA-CNN) focuses on the local patches, while Global Attention based CNN (GA-CNN) learns the global patterns from the entire facial region. To capture the complementary information between the two branches, Local–Global Attention-based CNN (LGA-CNN) is proposed. After feature aggregation, WSPP is used to learn the depression patterns. Comprehensive experiments on AVEC2013 and AVEC2014 depression databases have demonstrated that the proposed method is capable of mining the underlying depression patterns of facial videos and outperforms the most of the state-of-the-art video-based depression recognition approaches.