Evolving to multi-modal knowledge graphs for engineering design: state-of-the-art and future challenges

With the support of advanced information and communication technologies and open innovative design platforms, the emerging and blooming paradigm of mass personalization drives the process of engineering design to include knowledge with higher heterogeneity and more complex modalities. To this end, Multi-Modal Knowledge Graphs (MMKG), evolved from semantic networks and knowledge graphs, provide a powerful technology system for more effectively organizing and utilizing this knowledge. To understand the state-of-the-art of key aspects that enables MMKG, and recognize the potential challenges for forefront applications in engineering design, a literature review of MMKG-related publications is conducted. With selected 131 representative papers together with other 32 supplementary studies (up to 11/11/2023), this article summarizes the technical and practical efforts of multi-modal knowledge extraction, fusion technology, and specific applications in the engineering design process. Meantime, the challenges that MMKG faces and its foreseeable development potentials are discussed, which is hoped to provide a basis for the futuristic explorations and implementations of MMKG-enhanced availability and productivity in engineering design.