Fusing topology contexts and logical rules in language models for knowledge graph completion

Knowledge graph completion (KGC) aims to infer missing facts based on the observed ones, which is significant for many downstream applications. Given the success of deep learning and pre-trained language models (LMs), some LM-based methods are proposed for the KGC task. However, most of them focus on modeling the text of fact triples and ignore the deeper semantic information (e.g., topology contexts and logical rules) that is significant for KG modeling. For such a reason, we propose a unified framework FTL-LM to Fuse Topology contexts and Logical rules in Language Models for KGC, which mainly contains a novel path-based method for topology contexts learning and a variational expectation–maximization (EM) algorithm for soft logical rule distilling. The former utilizes a heterogeneous random-walk to generate topology paths and further reasoning paths that can represent topology contexts implicitly and can be modeled by a LM explicitly. The strategies of mask language modeling and contrastive path learning are introduced to model these topology contexts. The latter implicitly fuses logical rules by a variational EM algorithm with two LMs. Specifically, in the E-step, the triple LM is updated under the supervision of observed triples and valid hidden triples verified by the fixed rule LM. And in the M-step, we fix the triple LM and fine-tune the rule LM to update logical rules. Experiments on three common KGC datasets demonstrate the superiority of the proposed FTL-LM, e.g., it achieves 2.1% and 3.1% [email protected] improvement over the state-of-the-art LM-based model LP-BERT in the WN18RR and FB15k-237, respectively.