Color Facial Expression Recognition by Quaternion Convolutional Neural Network With Gabor Attention

Facial expression recognition (FER) plays an important role in cognitive psychology research. In FER studies, deep convolutional neural networks (CNNs) and attention mechanisms have shown great advantages over traditional techniques. Although many attention-based CNN models have been developed, most of them are proposed to deal with grayscale images. For color images, these network models usually take their grayscale versions as the input or convert them to grayscale data by the first hidden layer. Furthermore, their attention modules focus on only grayscale information. This mechanism does not fully consider the correlation between color channels, which will degrade the performance of color FER. To alleviate these shortcomings, we introduce quaternion techniques to CNNs and propose a quaternion CNN integrated with an attention mechanism (QA-CNN) for color FER. The proposed QA-CNN takes into account not only the correlation between color channels but also expressional attention. The new attention mechanism is based on a multidirectional quaternion Gabor filter. Besides, the proposed QA-CNN reduces the number of network parameters by 75% compared with the real-valued CNN with the same structure. Experimental results on three widely used data sets demonstrate the effectiveness of the proposed QA-CNN by showing clear performance improvements over other state-of-the-art FER methods.