Adaptive Language-Interacted Hyper-Modality Representation for Multimodal Sentiment Analysis

In an attempt to mitigate the problem of neglecting unimodal information and incorporating emotionally unrelated data during the fusion process of multimodal representation, this study presents an adaptive language interaction representation (Adaptive Language-interacted Representation, ALR) model in this study. Initially, the unimodal representation module is utilized to obtain a minimal but adequate representation of the unimodal information. Subsequently, we acknowledge that video and audio modalities may contain sentiment data that is not relevant. To address this issue, hyper-modality representation is constructed to mute the impact of irrelevant sentimental information. This is achieved through interaction among text, video and audio features. Finally, the hyper-modality representation is integrated through multimodal fusion module, harnessing more efficient multimodal sentiment analysis. On the datasets CMU-MOSEI, MELD and IEMOCAP, the model outperforms the major of existing sentiment analysis models.