Bidirectional Distillation for Top-K Recommender System

Recommender systems (RS) have started to employ knowledge distillation, which\nis a model compression technique training a compact model (student) with the\nknowledge transferred from a cumbersome model (teacher). The state-of-the-art\nmethods rely on unidirectional distillation transferring the knowledge only\nfrom the teacher to the student, with an underlying assumption that the teacher\nis always superior to the student. However, we demonstrate that the student\nperforms better than the teacher on a significant proportion of the test set,\nespecially for RS. Based on this observation, we propose Bidirectional\nDistillation (BD) framework whereby both the teacher and the student\ncollaboratively improve with each other. Specifically, each model is trained\nwith the distillation loss that makes to follow the other's prediction along\nwith its original loss function. For effective bidirectional distillation, we\npropose rank discrepancy-aware sampling scheme to distill only the informative\nknowledge that can fully enhance each other. The proposed scheme is designed to\neffectively cope with a large performance gap between the teacher and the\nstudent. Trained in the bidirectional way, it turns out that both the teacher\nand the student are significantly improved compared to when being trained\nseparately. Our extensive experiments on real-world datasets show that our\nproposed framework consistently outperforms the state-of-the-art competitors.\nWe also provide analyses for an in-depth understanding of BD and ablation\nstudies to verify the effectiveness of each proposed component.\n