Adaptive Multimodal Semantic Balancing Framework for Sentiment Analysis

Multimodal sentiment analysis is crucial in the existing research area. However, existing methods leverage the same static multimodal fusion order to deal with all samples, which ignores the impact of the sample variability issue. This fails to dynamically balance the sentiment semantics among multiple modalities for distinct samples. That is, the fixed modality fusion order may not be optimal for all types of samples, resulting in the suboptimal sentiment analysis. In this paper, we proposed an adaptive multimodal semantic balancing framework (MSBF) for sentiment analysis task. The above architecture consists of two essential part, the gated residual attention mechanism and mini-batch design. Specifically, the gated residual self-attention mechanism is proposed to investigate the significance and contribution of each modality, leading to the specific modality fusion order. Then, the gated residual cross-modality attention mechanism is utilized to integrate multiple modalities in a hierarchical manner. This allows us to effectively eliminate the semantic bias to some extend and adaptively balance the sentiment semantics according to the corresponding fusion order. Moreover, the mini-batch based strategy is leveraged to further handle the sample variability and facilitate the robustness of the multimodal semantic balancing procedure. The proposed adaptive multimodal semantic balancing design may provide the novel analysis paradigm for the multimodal learning and sentiment analysis. To evaluate the performance of the proposed model, the public multimodal sentiment analysis benchmark was introduced to perform the corresponding experiments. The experiments demonstrated that our model can obtain state-of-the-art performance.