Enhancing Multimodal Named Entity Recognition via Entity Span Detection and Graph Networks

Multimodal Named Entity Recognition (MNER) focuses on utilizing multiple modalities, including text and images, to recognize specific types of entities within text. Despite recent advancements, existing methods still face several challenges: (1) How to create a unified representation that connects and integrates different modalities, and (2) how to better interact between different modalities based on the unified representation while fully capturing both intra-modal and inter-modal information. In order to address these challenges, we design a new model, enhancing multimodal named entity recognition via entity span detection and graph networks (GENT-Net), which leverages entity recognition and graph networks to enhance MNER. Specifically, we employ a graph-based structure to model text and images, use a graph attention network for multimodal fusion, and introduce entity span detection using a graph convolutional network to assist the MNER task. We perform comprehensive experiments on the publicly available datasets, Twitter2015 and Twitter2017, showcasing the effectiveness of our proposed approach.