Image-text Fusion Model for Depression Tendency Detection Based on Attention

Social media information can be used for monitoring multiple mental health issues, including depression. Aiming to detect users with depression tendency on social network, this paper proposes a depression tendency detection model, using image and text posts by Sina Weibo users. In this model, ALBERT and VGG16 are applied to extract text and image features, respectively. Considering that the information in short text is limited, the model utilizes Biterm Topic Model (BTM) to obtain topic distribution, then concatenates it into the original text. In addition, the paper proposed an image-text fusion method based on attention mechanism, which enabled text and images to interact. Finally, the fused features are applied to the task of depression detection. Experimental results indicate that the proposed model outperforms the unimodal detection model in terms of precision, recall and F1 score, and the fusion method based on the attention mechanism is better than concatenating.