Hierarchical self-adaptation network for multimodal named entity recognition in social media

Abstract Multimodal Named Entity Recognition task aims to identify named entities in user-generated posts containing both images and texts. Previous multimodal named entity recognition methods greatly benefit from visual features when the text and the image are well aligned, but this is not always the case in social media. On condition that the image is missing or mismatched with the text, these models usually fail to provide excellent performance. Besides, previous models use only single attention to capture the semantic interaction between different modalities, which largely ignore the existence of multiple entity objects in images and texts of the posts. To alleviate these issues, we present a novel model named Hierarchical Self-adaptation Network (HSN) to address these issues. The HSN contains 1) a Cross-modal Interaction Module to promote semantic interaction for the multiple entity objects in different modalities, which is proved to suppress wrong or incomplete attention in multimodal interactivity; 2) a Self-adaptive Multimodal Integration module to handle the problems that the images are missing or mismatched with the texts. Additionally, to evaluate the adaptability of HSN in real-life social media, we construct a Real-world NER dataset consisting of plain text posts and multimodal posts from Twitter. Extensive experiments demonstrate that our model achieves state-of-the-art results on the Real-world multimodal NER dataset and the Twitter multimodal NER dataset.