Gradient Embedding and Modal Complementarity Networks for Audio-Video Depression Level Prediction

Physiological studies show distinct audio-video behaviors correlate with depression levels, leading some researchers to use Transformer architecture to predict depression levels. The positional embedding in Transformer assigns a positional marker to each element, but the multi-head attention mechanism operates at the token level and is difficult to examine the temporal changes of elements. This brings limitations in capturing dynamic differences reflecting depression. Besides, previous works adopt element-wise operation, fully connected layers or attention mechanisms for modal complementarity. However, they don't specify what the supplemental representation is, and don't explore to what extent the supplemental representation is accepted. This hinders the model to characterize depression cues. Therefore, we propose the Gradient Embedding (GE) and the Modal Complementarity (MC) modules. The GE module embeds the gradient into sequences to perceive the dynamics of elements, thereby helping to capture depression-related temporal patterns. Besides, we stipulate the supplementary representation as the difference between two modal representations. In this way, the MC module quantifies to what extent the supplemental representation is accepted by a weighted combination of unimodal representation and supplemental representation, thereby achieving effective modality complementarity. Experimental results (RMSE = 6.42/6.23, MSE = 5.06/5.00) on AVEC 2013 and AVEC 2014 datasets illustrate the progressiveness of our method.