Sequence Knowledge Enhancement Distillation Framework for Ultra-Fast Image Deraining

Traditional knowledge distillation techniques are aimed at compressing models and speeding up inference, but they often fail to maintain the superior capabilities of complex models in simpler ones. To address this issue, this paper focus on the deraining task and introduces the Sequential Knowledge-Enhanced Distillation Framework (SKEDF). SKEDF, as a two-stage strategy, comprises a Knowledge Completion Stage (KCS) and a Knowledge Enhancement Stage (KES). The KCS employs feature sequence to enhance the student network’s ability to understand and learn superior deraining capabilities from various teacher networks. The KES independently trains the student network to further refine its deraining abilities. Moreover, the framework incorporates a Contrastive Structural Similarity Regularization (CSSR) loss, ingeniously integrating contrastive learning with the Structural Similarity Index (SSIM) to enhance training efficacy and achieve nuanced model improvements. Quantitative and qualitative results demonstrate that SKEDF not only achieves a breakthrough improvement in model efficiency but also delivers more promising deraining performance compared to other SOTA solutions.