Personalized multimodal sentiment analysis under uncertain modalities missing via pretraining and online learning

• We proposed the problem of personalized MSA for the first time, and develop an effective model via pre-training and online learning methods. • To effectively to handle the uncertain missing modalities, we propose to reuse the fused modalities to complete the missing modalities. • We propose a set of online learning methods to enable the pretrained MSA model to autonomously adapt to personalized users. • We conduct extensive experiments to verify the performance of our proposed model with three public benchmark datasets (IEMOCAP, MELD, CMU-MOSI). Currently, multimodal sentiment analysis (MSA) for personalized users under uncertain modalities missing has become a new challenging problem. To address this issue, we propose a two-step idea. First, we propose an effective MSA model under uncertain modalities missing and train it with some public datasets, thus to enable the model to possess better preliminary MSA ability. Then, we make the pretrained model to continuously learn user’s personalized characteristics with online learning methods, thereby enable the model grow into a robust model for personalized MSA. Based on this idea, we propose a Personalized MSA model under uncertain modalities missing via Pretraining and Online Learning (termed as PMSAPO). For Personalized MSA under uncertain modalities missing, PMSAPO firstly generates the fused modality and allocate weights for each modality with a Fully Connected Neural Network Evaluation Module. Then, PMSAPO completes the final sentiment classification based on the fusion modality with a Joint feature optimization module. For the pretrained PMSAPO, we make it autonomously learn the personalized users via our proposed online learning techniques, including an online meta-learning method, a learning rate adaptive adjustment strategy, and a dynamic weight assignment strategy for sample data. Finally, based on three public benchmark datasets (IEMOCAP, MELD and CMU-MOSI), we conduct extensive experiments and prove that PMSAPO completely outperforms the Twelve state-of-the-art baseline models. (Code is available at https://github.com/SHX-AI/PMSAPO .)