Identity and Modality Attributes Driven Multimodal Fusion Networks for Emotion Recognition in Conversations

Emotion recognition in conversations (ERC) is a crucial aspect of human-computer interaction and plays an important role in various domains, including healthcare, entertainment, and education. Since the conversation data in the form of multimodal sequences is well suited to be constructed into graphs, the methods based on graph convolutional network (GCN) show incomparable advantages. However, existing methods attempt to model the highly uncertain emotional relationships between different speakers, which is not an easy task and may even introduce interference information. Therefore, we propose an identity and modality attributes driven multimodality fusion network (dubbed IMDNet) for emotion recognition in conversations. Specifically, we construct a speaker-centric graph that only connects nodes of the same speaker within modalities to each other, reducing the interference between the emotions of different speakers. We also introduce the attribute embedding mechanism, which facilitates the correct calculation of correlations between nodes for better multimodal feature fusion. Considering that the emotional correlation between utterances will decrease over time, we present an utterance distance attention to make the fusion network pay more attention to the adjacent utterances. Furthermore, we explore the solution to the data imbalance problem suitable for conversation scenarios. Given the presence of possible anomalous samples in the dataset, we opt for the BoundaryFocalLoss. Experiments on the IEMOCAP and MELD datasets show that our IMDNet outperforms the state-of-the-art methods.