MLM-EOE: Automatic Depression Detection via Sentimental Annotation and Multi-Expert Ensemble

Automatic Depression Detection (ADD) advances rapidly, though challenges persist. First, medical studies show distinct emotional changes between individuals with depression and those without, yet few studies effectively leverage emotions for ADD. Many existing models use a single processing module across various emotions, but the data characteristics differ under different emotions. A single module may not be able to grasp these data characteristics simultaneously, leading to the loss of key depressive clues. Second, fixed scales and perspectives are commonly applied in feature modeling, though depressive states and physiological signals change dynamically. Fixed modeling approaches may truncate or misconnect depressive signals. This study proposes a Multimodal Large Model-driven Ensemble of Expert Networks (MLM-EOE) for ADD, consisting of three components: (1) Multimodal Large Model Sentiment Annotation (MLM-SA) to annotate emotions in raw video data; (2) Ensemble of Experts (EOE) to capture data features under various emotional states; and (3) Modeling Inter and Intra Multiscale Patches (MIIMP) to apply multiscale, multi-perspective modeling to expert-processed data. Extensive experiments on four datasets (AVEC2013, AVEC2014, AVEC2019, and CMDep) validate the effectiveness of MLM-EOE. The self-constructed CMDep dataset is available upon request via the provided link. The code is publicly available at https://github.com/ZulongLin/MLM-EOE.