CRGMR: A Contextualized RGAT and GraphTransformer Method for Multimodal Emotion Recognition

The recognition of emotions in dialogue is crucial for building empathetic machines. Current research primarily focuses on learning emotion-related common features in multimodal data, neglecting the complex dependency relationships of emotional features within dialogues (local and global information). To address this issue, we propose A Contextualized RGAT and GraphTransformer Method for Multimodal Emotion Recognition (CRGMR) that leverages both local information (dependencies between speakers) and global information (context of the dialogue) in the conversation. This model utilizes Transformer to capture cross-modal global information, constructing a heterogeneous graph with extracted global features. Then, it employs RGAT and GraphTransformer to model both global and local information within the heterogeneous graph. Extensive experiments demonstrate the effectiveness of CRGMR via its state-of-the-art results on the IEMOCAP and MELD datasets for the multimodal ERC task.