Reward-Based Gradient Modulation for Multimodal Emotion Recognition With LoRA

Multimodal sentiment analysis (MSA) has emerged as a prominent research area in the field of computer science, focusing on the comprehension of human behaviors by computational systems. Most existing pipelines usually involve two steps: unimodal representation and multimodal fusion. However, on one hand, within this process, the varying contributions of different modalities can cause imbalance in multimodal training. On the other hand, in unimodal representation, the inclusion of pretrained models presents challenges of slow training and difficult optimization. Based on the aforementioned findings, we have redefined the workflow for current multimodal emotion recognition. In this study, we introduce a novel approach called reward-based gradient modulation for regulating the convergence speed of individual modalities within the fusion network with LoRA (RGM-LoRA), aimed at achieving a balanced integration process. Additionally, text modality encoders commonly employ large-scale language pretraining models such as BERT. We introduce LoRA to alleviate the problem of text modality optimization being suppressed by the other two modalities. To our knowledge, we are the pioneering researchers who have provided evidence showcasing the efficacy of LoRA in the context of optimization. Finally, to further ensure the effect of textual modality, we add intermodal contrast learning. As a result, we achieve the state-of-the-art (SOTA) performance on two public benchmark datasets, CMU-MOSI, and CMU-MOSEL.