KD-BERT: A Lightweight Knowledge Distillation Bidirectional Encoder Representations from Transformers for IoT Network Intrusion Detection

The Internet of Things (IoT) network intrusion detection (NID) has emerged as a powerful technique for effectively detecting attacks and protecting internet safety across a wide range of applications. However, existing NID models often struggle to achieve good detection performance when deployed in resource-restricted environments. To achieve an optimal balance between detection performance and computational cost, we propose a lightweight knowledge distillation bidirectional encoder representations from Transformers (KD-BERT) framework for efficient IoT intrusion detection. Specifically, we first develop a header-payload pairs tokenization method to preprocess the raw data. Then, we distill rich knowledge from a large pre-trained $\text{BERT}_{\text{teacher}}$ model into a smaller $\text{BERT}_{\text{student}}$ model with fewer training parameters. By leveraging KD-BERT, the $\text{BERT}_{\text{teacher}}$ can be compressed into the $\text{BERT}_{\text{student}}$, reducing the number of parameters from 135 to 9 M, while maintaining high detection accuracy. To effectively obtain the essential features, we design a data transformation strategy and fine-tuned the $\text{BERT}_{\text{student}}$ model on several labeled IoT NID tasks. Comprehensive experimental results on three public datasets (telemetry datasets, operating systems datasets and network traffic dataset of IoT (TON-IoT), Edge-IIoTset, and IoMT2024) have demonstrated superior performance compared to state-of-the-art methods.