From Emotion to Depression: How to leverage Emotional Representations for Multimodal Depression Diagnosis?

Existing depression diagnosis methods attempt to understand depression via general emotional representations learning and have achieved notable progress. However, these methods still remain limited in two key aspects: 1) coarse-grained emotional representation: these methods often struggle to capture fine-grained multimodal emotional cues; 2) representation gap: general emotional knowledge alone is insufficient to fully account for the complex and multifaceted depression symptom. To address these limitations, we propose a hierarchical multimodal knowledge injection framework named DEmopression, comprising multimodal emotion knowledge injection and depression knowledge injection stages that integrate fine-grained emotion representations with specific depression knowledge for multimodal depression recognition. In the multimodal emotion knowledge injection stage, we leverage a multimodal large language model to generate fine-grained emotional prior knowledge and integrate it into the multimodal representation learning process through text-bridge contrastive learning. In the depression knowledge injection stage, we construct a depression knowledge graph and employ a cross-attention injection module that enables the DEmopression to adaptively perceive relevant depressive symptoms, thereby bridging the representation gap between emotion understanding and depression cognition. Experimental evaluations on multiple public datasets demonstrate that DEmopression achieves state-of-the-art performance in depression diagnosis. In particular, compared with previous leading approaches, DEmopression reduces the MAE by 6.4% and the RMSE by 2.8% on the AVEC2014 dataset. Moreover, this study provides a comprehensive and in-depth analysis of DEmopression, examining its capability to perceive individual-level depressive symptoms and offering valuable insights for future research on depression diagnosis. Our code will be released after acceptance.