Multimodal Conversational Emotion Recognition in Education With Uncertain Missing Modalities: A Graph Fusion Network Approach

Multimodal Emotion Recognition in Conversations (MERC) under uncertain missing modalities remains a critical challenge, especially in educational contexts. Significant research efforts have been dedicated to developing novel solutions to tackle this problem. Nevertheless, existing methods are difficult to fully utilize available information for high-quality data reconstruction, and fail to learn discriminative multimodal representations for handling uncertain missing modalities. Furthermore, the generalization of such methods in educational contexts remains largely overlooked. To address these challenges, this study proposes a novel graph fusion network approach for achieving MERC tasks in education with uncertain missing modalities. Our method first reconstructs missing modalities by using adaptive fusion weights to aggregate information from the most similar neighbors in the available modalities. The reconstructed data is processed via a mask-aware graph structure to explicitly exploit masked cues to enhance the robustness of intra-modal representations. Initial multimodal fusion representations derived from single-modal features are employed to build a dual-graph structure for further refining the multimodal representation learning of each utterance. During the training process, our method also introduces the boundary-aware contrastive learning to enable the accurate identification of samples near decision boundaries. Experiments on three multimodal datasets, including one self-collected educational dataset, CMU-MOSI, and CMU-MOSEI, demonstrates its superiority, effectiveness, and generalizability.