Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion

Multimodal Knowledge Graphs (MKGs), which organize visual-text factual\nknowledge, have recently been successfully applied to tasks such as information\nretrieval, question answering, and recommendation system. Since most MKGs are\nfar from complete, extensive knowledge graph completion studies have been\nproposed focusing on the multimodal entity, relation extraction and link\nprediction. However, different tasks and modalities require changes to the\nmodel architecture, and not all images/objects are relevant to text input,\nwhich hinders the applicability to diverse real-world scenarios. In this paper,\nwe propose a hybrid transformer with multi-level fusion to address those\nissues. Specifically, we leverage a hybrid transformer architecture with\nunified input-output for diverse multimodal knowledge graph completion tasks.\nMoreover, we propose multi-level fusion, which integrates visual and text\nrepresentation via coarse-grained prefix-guided interaction and fine-grained\ncorrelation-aware fusion modules. We conduct extensive experiments to validate\nthat our MKGformer can obtain SOTA performance on four datasets of multimodal\nlink prediction, multimodal RE, and multimodal NER. Code is available in\nhttps://github.com/zjunlp/MKGformer.\n