Atlantis: Aesthetic-oriented multiple granularities fusion network for joint multimodal aspect-based sentiment analysis

Joint Multi-modal Aspect-based Sentiment Analysis (JMASA) is a challenging task that seeks to identify all aspect-sentiment pairs from multimodal data. Current JMASA studies are insufficient in bridging the representational gap between textual and visual modalities. Additionally, they largely emphasize image feature extraction, neglecting the exploration of image presentation forms, like aesthetic characteristics. In this paper, we propose an Aesthetic-oriented Multiple Granularities Fusion Network for JMASA, termed Atlantis. This trident-shaped framework comprises three branches: Textual-vision Alignment Aspect-sentiment Extraction, Sentiment-aware Image Aesthetic Assessment, and Aesthetic-aware JMASA. Notably, the first two branches function as auxiliary learning tasks, with Textual-vision Alignment Aspect-sentiment Extraction aimed at bridging the representational gap between modalities, and Sentiment-aware Image Aesthetic Assessment dedicated to understanding the aesthetic attributes of images. Concurrently, the Aesthetic-aware JMASA dynamically integrates varied granular features from both branches to perform JMASA. To the best of our knowledge, this is the first aesthetic-oriented approach in the present field. Experimental results on two public datasets verify that Atlantis outperforms a series of prior strong methodologies and achieves a new state-of-the-art (SOTA) performance. The enhancement highlights Atlantis’s advanced capability in accurately identifying aspect-sentiment pairs with aesthetic features.