计算机科学
人工智能
模态(人机交互)
校准
融合
情绪分析
人工神经网络
传感器融合
模式识别(心理学)
信息融合
情感计算
机器学习
情绪识别
语音识别
计算机视觉
自然语言处理
接头(建筑物)
深度学习
数据挖掘
作者
Yuxuan Hu,Yushan Pan,Xianfei He,Xinfei Guo,Hao Wang,Kaiwei Wang,Yanlong Cao,Zhijie Xu
标识
DOI:10.1109/taffc.2026.3726067
摘要
Multimodal sentiment analysis (MSA) aims to model the temporal evolution of affective states by jointly analyzing textual, visual, and acoustic modalities. Despite recent progress, existing methods remain constrained by several fundamental limitations: they often rely on a static modality importance assumption, preventing adaptation to evolving affective expressions; they lack explicit modeling of temporal anomalies such as sentiment transition points and cross-modality phase differences, leading to unreliable fusion decisions at critical moments; and their uniform treatment of temporal contexts ignores the intrinsic temporal scales of different modalities. To overcome these limitations, we propose the Temporal Calibration Network (TCaN), which reformulates multimodal fusion as a temporally calibrated fusion-guidance process. In this formulation, the guidance signal refers to the explicit estimation and calibration of time-varying modality salience before cross-modal feature interaction, rather than post-hoc calibration of final prediction logits. TCaN introduces a Tempo-Gated Router (TGR) that performs modality-specific temporal encoding with tailored history windows, enabling dynamic routing of modality importance over time. To explicitly address temporal anomalies, a Temporal Synergy Calibrator (TSC) jointly models transition intensity and temporal alignment through a learnable synergistic gating mechanism, ensuring robust decisions during affective transitions. The resulting calibrated guidance further guides a Calibrated Guidance Fusion (CGF) module for temporally coherent cross-modal interaction. Extensive experiments on CMU-MOSI, CMU-MOSEI, CH-SIMS, and MIntRec demonstrate the effectiveness of TCaN, achieving Acc-7 of 52.39 on MOSI and 56.15 on MOSEI, and Acc-2 of 84.92 on CH-SIMS, with notable improvements in correlation and F1 scores. These results highlight TCaN's effectiveness in capturing high temporal affective dynamics and making robust cross-modal fusion decisions.
科研通智能强力驱动
Strongly Powered by AbleSci AI