An Ordinal Learning Framework with Sentiment Decouple for Multimodal Sentiment Analysis

Multimodal sentiment analysis seeks to effectively integrate information from acoustic, visual, and language modalities for the prediction of sentiment. Most previous studies perform regression based on annotated sentiment values, which often neglect the ordinal nature of sentiment. In this paper, we address this gap by exploring ordinal learning of sentiment through the development of ordinal margin losses and sentiment decouple module. Specifically, we propose cross-class and class-specific ordinal margin losses that encourage the model to learn to compare, incentivizing larger predictive values for samples annotated with larger sentiment values over those with smaller sentiment values. Moreover, a sentiment decouple module is innovatively designed to decouple sentiment prediction into polarity prediction and intensity prediction tasks, reducing the difficulty of sentiment prediction. We also dynamically maintain a high-quality baseline embedding for neutral samples, which allows the model to perform ordinal learning during sentiment prediction by calculating the difference between the baseline embedding and the features of individual samples. The experimental results suggest that the proposed method outperforms state-of-the-art approaches across three datasets, with a more than 6.6% decrease in the MAE on the CMU-MOSI dataset.