SupGATE: Supervised Contrastive Learning and Attention-Gated Multimodal Fusion for Emotion Recognition in Conversations

Emotion Recognition in Conversations (ERC) is vital for developing empathetic and responsive systems that accurately understand and react to users’ emotional states. While traditional ERC methods primarily rely on textual data, incorporating multimodal information—such as audio and visual cues—can significantly improve its performance by capturing complementary emotional signals. However, effectively leveraging these diverse modalities remains challenging due to the inherent heterogeneity of multimodal data. Features from different modalities often reside in distinct semantic spaces and noisy or less relevant signals from one modality may interfere with the extraction of meaningful emotional cues from others. In this paper, we propose Supervised Contrastive Learning and Attention-Gated Multimodal Fusion (SupGATE), a novel framework designed to enhance multimodal ERC by addressing modality alignment and fusion. SupGATE utilizes supervised contrastive learning (SCL) to align features across textual, acoustic, and visual modalities, ensuring that representations of the same emotion are closely clustered while those of different emotions are distinctly separated. Additionally, our Attention-Guided Gating Module (AGGM) dynamically assigns weights to each modality during the fusion process, enabling the model to prioritize the most informative emotional signals. To further enrich the model’s understanding of emotional dynamics, SupGATE incorporates an auxiliary task that predicts the emotional transition in the subsequent conversational turn, capturing the evolving emotional context within multi-turn and multi-party dialogues. Finally, We evaluate SupGATE on two benchmark datasets, IEMOCAP and MELD, achieving state-of-the-art performance on MELD and competitive results on IEMOCAP.