Video-Based Depression Level Analysis by Encoding Deep Spatiotemporal Features

As a serious mood disorder problem, depression causes severe symptoms that affect how people feel, think, and handle daily activities, such as sleeping, eating, or working. In this paper, a novel framework is proposed to estimate the Beck Depression Inventory II (BDI-II) values from video data, which uses a 3D convolutional neural network to automatically learn the spatiotemporal features at two different scales of the face regions. Then, a Recurrent Neural Network (RNN) is used to learn further from the sequence of the spatiotemporal information. This formulation, called RNN-C3D, can model the local and global spatiotemporal information from consecutive face expressions, in order to predict the depression levels. Experiments on the AVEC2013 and AVEC2014 depression datasets show that our proposed approach is promising, when compared to the state-of-the-art visual-based depression analysis methods.