Geo‐ MAG : A Knowledge Graph ( KG )‐enhanced Multimodal Retrieval‐Augmented Generation ( RAG ) Framework for Geological Map Understanding

ABSTRACT The geological reports and maps accumulated during geological surveying and mapping harbor rich expert knowledge and metallogenic clues. However, efficiently integrating and mining structured knowledge from complex multimodal data of polymetallic deposits remains a critical bottleneck in intelligent mineral prediction. To address this, we propose a knowledge graph (KG)‐enhanced multimodal retrieval‐augmented generation (RAG) framework, Geo‐MAG, for geological map understanding. Specifically, the framework first processes textual geological reports and constructs a structured KG. Concurrently, a vision large model parses geological maps to extract metadata, including legends, geological structures, strata, and lithologies. Leveraging this metadata, relevant subgraphs are retrieved from the KG to facilitate text–map semantic alignment and enhance background geological knowledge. Finally, the integrated map information and structured subgraphs of KG are fed into the GPT‐4o to enable deep semantic interpretation. Experimental results demonstrate that integrating the knowledge graph significantly boosts the GPT‐4o's reasoning capability and interpretability in geological map understanding. The model achieves 77.2% accuracy in geological reasoning tasks, outperforming the direct end‐to‐end GPT‐4o interpretation by 53.7% and lightweight schemes on the basis of basic metadata by 37.4%. This work represents a pioneering application of KG and RAG in geological map understanding, highlighting the synergistic advantages of integrating text and maps, and offering a novel perspective on multimodal integration within the geoscience domain.