Knowledge Distillation-Based Spiking Neural Network for Online Video Action Understanding

Online action detection and anticipation aim to understand current or upcoming actions in video streams. In industry, current artificial neural network (ANN)-based methods suffer from prohibitive energy consumption, fundamentally limiting their deployment on resource-constrained industrial devices. To bridge the gap between theory and application practice of informatics in industrial environments, we propose a novel knowledge distillation-based spiking neural network (KDSNN), which synergistically integrates bioinspired spike-driven processing with knowledge distillation, significantly reducing the energy consumption. Specifically, KDSNN includes a pioneering spiking neural network (SNN) architecture for online action detection and anticipation, which combines well-designed hierarchical spike convolutional neural network (CNN) block and spike Transformer block to capture spike-driven information. To further improve the performance of our SNN while maintaining low energy consumption, we introduce the knowledge distillation paradigm, which aims to utilize an expert-level ANN as a teacher to guide our SNN. Based on this, we propose a novel distillation loss, which consists of feature distillation and logit distillation. Notably, to address the cross-domain feature alignment in feature distillation, the optimal transport theory is employed to realize cross-domain knowledge transfer for the first time by minimizing the Wasserstein distance between continuous features (ANNs) and discrete features (SNNs). Through extensive evaluations on THUMOS14 and EPIC-Kitchen-100 datasets, the energy consumption of our KDSNN is only 27.1% and 10.0% of the state-of-the-art ANN-based method MAT. Equally importantly, the parameter count of our KDSNN is only 37.0% and 27.7% of MAT on THUMOS14 and EPIC-Kitchen-100, respectively.