Multimodal Sentiment Analysis via Low-Rank Tensor Attention Network with Unimodal Self-Supervised Learning

Despite the substantial advancements made in multimodal sentiment analysis, there remains scope for enhancing multimodal accuracy through improved fusion techniques. The challenge in multimodal sentiment analysis lies in the inadequate utilization of deep potential correlations between modalities and the proper management of modality consistency and difference, particularly when incorporating fusion methods with unified multimodal labels. In the paper, we propose a novel approach called Low-Rank Tensor Attention Network with unimodal self-supervised learning. In this method, the Low-Rank Tensor Attention Network effectively captures the relevant information of low-level features, and reveals the underlying correlation among different modalities. Moreover, by leveraging the parallel decompositions of the low-rank weight tensor and the attention mechanism, cross-modal complementary semantic information can be captured by interacting with all modalities simultaneously. In addition, we utilize unimodal sentiment labels in a self-supervised manner to guide multimodal and unimodal tasks, allowing us to obtain commonalities and characteristics across different modalities. Our proposed method demonstrates competitive performance on the MOSI, MOSEI, and SIMS datasets, outperforming current state-of-the-art methods on most metrics across the publicly available datasets.