情绪分析
计算机科学
多式联运
模式治疗法
人工智能
心理学
业务
营销
心理治疗师
作者
Peizhu Gong,Jin Liu,Xiliang Zhang,Xingye Li,Lai Wei,Huihua He
出处
期刊:
[Institute of Electrical and Electronics Engineers]
日期:2024-11-27
卷期号:33: 23-36
被引量:10
标识
DOI:10.1109/taslp.2024.3507576
摘要
Most current models for analyzing multimodal sequences often disregard the imbalanced contributions of individual modal representations caused by varying information densities, as well as the inherent multi-relational interactions across distinct modalities. Consequently, a biased understanding of the intricate interplay among modalities may be fostered, limiting prediction accuracy and effectiveness. To address these critical issues, we propose the Adaptive Multimodal Graph Integration Network (AMGIN) for multimodal sentiment analysis. Concretely, AMGIN transforms multimodal sequences into graph structures and discriminatively fuses complex intra- and inter-modal correlations by incorporating multiple edge types. To accurately modulate each modality's contribution, we present the Adaptive Modality Adjustment Mechanism (AMAM), consisting of two primary components: a modality refinement loss (MR Loss) that selectively optimizes the parameters of unimodal branches through backpropagation according to the relative confidence of modalities, and a modality-confidence gated module (MCGM) that adaptively filters out noise according to the deviation of modality-specific representations from the shared semantic distribution. Furthermore, we introduce feature reconstruction loss as an additional constraint to prevent excessive modification. To verify the effectiveness of our proposed approach, extensive experiments are conducted on three benchmark datasets commonly used in sentiment analysis, namely IEMOCAP, CMU-MOSEI, and CMU-MOSI. Additionally, we consider the task of multimodal humor detection on the UR-FUNNY dataset. The experimental results demonstrate the superiority of AMGIN over the state-of-the-art methods.
科研通智能强力驱动
Strongly Powered by AbleSci AI