Learning Three-Modal Joint Representations With Multimodal Transformer for Multimodal Sentiment Analysis

Understanding user sentiment from multiple modalities is essential for applications such as personalized recommendations and affective computing. We propose TMJR, a Three-modal Joint Representation model for multimodal sentiment analysis, which introduces a structured three-round cross-modal attention mechanism to integrate textual, acoustic, and visual information progressively. Unlike prior one-pass or pairwise fusion methods, TMJR models multiple directional interaction paths to capture deeper cross-modal dependencies and improve semantic alignment. We evaluate TMJR on two benchmark datasets: CMU-MOSI and CMU-MOSEI. It achieves state-of-the-art results, with an F1 score of 86.4% and a Pearson correlation of 0.799 on MOSI, and an F1 score of 86.5% and a correlation of 0.775 on MOSEI. Further ablation, label-efficiency, and visualization analyses confirm the model’s robustness. TMJR provides a promising solution for sentiment-aware recommendation and multimodal user understanding.