Data Redundancy Detection Algorithm based on Multidimensional Similarity

In order to address the problem of misjudgment and missing judgment in duplicate data detection by traditional similarity method. The multidimensional similarity redundancy detection algorithm MSRD is proposed in this paper, which combines numerical similarity, literal similarity, and semantic similarity to form a more complete similarity calculation method, and the detection process is constructed based on the idea of priority queue. In this paper, four real and synthetic data sets and three traditional redundancy detection tools are compared. The results show that this method can better distinguish the similarity between records, and in the process of redundancy detection, it can ensure that the detection efficiency is similar to the traditional algorithm, with better precision and recall, and can more accurately detect similar duplicate data.