An Innovative Multisource Teacher Collaborative Framework for Self-Knowledge Distillation

Self-knowledge distillation, abbreviated as SKD, exhibits greater computational efficiency than traditional knowledge distillation (KD) because it learns from its own predictions rather than from a pretrained teacher. Existing SKD methods diversify knowledge through auxiliary branches, data augmentation, historical models, and label smoothing. However, previous methods primarily extract knowledge from a single-source teacher, overlooking the diversity and complementarity of various types of teacher knowledge in model learning, thereby limiting performance improvements. In response to this challenge, we propose a pioneering paradigm termed multisource teacher collaboration for self-knowledge distillation (MSTCS-KD), which integrates knowledge from diverse types of teachers to complementarily enhance the model's learning capability. We start by adding lightweight auxiliary branches with different structures in the shallow layers to build the student network, while also incorporating a teacher-guided attention mechanism to support adaptive learning. Then, we perform collaborative distillation by combining "heterogeneous knowledge" from the primary network's deepest layers with "homogeneous knowledge" from the student's outputs on augmented samples. This complementary distillation approach improves the model's ability to learn features, generalize, and enhance trainability. Extensive experiments demonstrate that our method outperforms other state-of-the-art SKD methods across various network architectures and datasets.