A multi-path fusion with knowledge augmentation framework for multimodal NER in Chinese painting

Chinese painting is an important form of cultural heritage. Its digital resources, including images and descriptive texts, contain rich yet unstructured information. Multimodal named entity recognition (MNER) is crucial for knowledge extraction but faces challenges including the lack of domain-specific datasets, limited integration of external knowledge, and semantic gaps between modalities. To address these issues, we construct CP-MNER, an MNER dataset for Chinese painting, and establish standardized baselines. We further propose MFKA, a multi-path fusion framework with knowledge augmentation. MFKA generates text-aware visual representations through cross-modal attention and incorporates external knowledge through a two-stage process using multimodal large language models. A multi-path complementary fusion module then integrates textual, visual, and knowledge-enhanced representations for multimodal semantic alignment. Experimental results demonstrate that MFKA achieves state-of-the-art performance on CP-MNER.