Mamba-Based Enhanced Multimodal Emotion Recognition with EEG Guidance

Multimodal emotion recognition integrates information from multiple modalities to more comprehensively capture emotional features, enhancing recognition accuracy. Although existing methods have achieved significant results, irrelevant or conflicting emotional information across modalities often limits performance improvement. Inspired by Mamba's ability to effectively filter irrelevant information and model long-range dependencies with linear complexity, we propose a new paradigm for Mamba-based enhanced EEG-guided multimodal emotion recognition (MambaMER). This paradigm effectively addresses the interference caused by cross-modal information conflicts, enhancing the performance of multimodal emotion recognition. Firstly, to alleviate the interference caused by conflicts between different modalities, we design a multi-scale EEG-guided conflict suppression module. In this module, guided by multi-scale EEG features, the selective cross state space model dynamically adjusts its parameters through a selective mechanism, effectively suppressing emotionally irrelevant or conflicting information within the eye movement modality, thereby obtaining enhanced eye movement features. Secondly, to deeply integrate the complementary features between the EEG modality and the enhanced eye movement modality, we propose a novel cross-modal fusion mechanism, consisting of Mutual-Cross-Mamba and Merge-Mamba. This mechanism not only effectively captures long-range dependencies within the fused features but also maintains linear computational complexity, thereby enhancing the efficiency of cross-modal information integration and utilization. Experimental results on the SEED, SEED-IV, and SEED-V datasets demonstrate that our method significantly surpasses current state-of-the-art methods.