Micro-expression recognition using 3D DenseNet fused Squeeze-and-Excitation Networks

Micro-expression is a kind of facial feature that reflects the most real emotional state hidden in the human heart. Most of the existing micro-expression recognition methods are based on manual feature extraction of subtle movements of facial muscles. Due to its short duration and weak intensity, the accurate identification of micro-expression remains a challenging task. This paper investigates micro-expression recognition based on deep learning methods and proposes a three-dimensional SE-DenseNet architecture, which fused Squeeze-and-Excitation Networks with a 3D DenseNet and can automatically integrate the spatiotemporal features extracted from each video to increase the weight of valid feature maps. The proposed architecture first obtains apex frames from each video for the most obvious facial muscle movements and then amplifies facial muscle movements using Euler video magnification to significantly alleviate the issue of small sample size and weak intensity of micro-expression recognition. Finally, the pre-processed videos are fed into the 3D SE-DenseNet for further feature extraction as well as to perform micro-expression classification. Experiments are performed on three public datasets. Our best model obtains an overall accuracy of 95.12%, 92.96%, and 82.74% on SMIC, CAS(ME) 2 and CASME-II dataset, respectively. The experimental results show that the proposed methods can well describe the considerable details of micro-expression and outperform most of the state-of-the-art methods on three public datasets. • Appropriate preprocessing promotes the extraction of micro-expression features. • The three-dimensional DenseNet can extract facial features deeply. • SE block combined with DenseNet can facilitate feature extraction. • Different SE block combination methods significantly affect the recognition rate.