Injecting Multimodal Information Into Pre-Trained Language Model for Multimodal Sentiment Analysis

With the increasing availability of computational and data resources, numerous powerful pre-trained language models (PLMs) have emerged for natural language processing tasks. However, how to inject nonverbal modalities into PLMs to handle multimodal information remains a practical problem. In this paper, we explore the application of PLM on multimodal sentiment analysis from a different perspective. Unlike many recent methods that develop multimodal fusion layers that are sequential to attention layers, we investigate the effectiveness of cross-modal additive attention that is parallel to attention layers, which takes the language modality as dominant modality. Moreover, we devise a gating mechanism to control the flow of nonverbal information by estimating its discriminative level. In this way, we can prevent noisy multimodal information from damaging the performance of pre-trained language model. In our framework, nonverbal modalities serve as auxiliary roles to provide the model with additional information and improve the understanding of multimodal human language. Additionally, cross-modal margin and matching losses are proposed to align the distributions of various modalities and simultaneously retain modality-specific information, which to some extent address the shortcoming of contrastive learning loss. Comprehensive experiments show that our approach surpasses existing state-of-the-art methods on multimodal sentiment analysis and emotion recognition tasks.