Multi-Stage Network Attack Detection Algorithm Based on Gaussian Mixture Hidden Markov Model and Transfer Learning

Multi-stage network attack (MSA) is a serious threat to data security. The high-dimensionality of the alert data along with the diverse features, leads to poor detection performance for MSA. Consequently, this paper proposes a multi-stage network attack detection algorithm based on Gaussian mixture hidden Markov model and transfer learning. Firstly, a sequence modeling framework of Gaussian mixture hidden Markov models is proposed. It uses a Gaussian mixture model to cluster high-dimensional alert data and a hidden Markov model to fully consider the temporal structure of MSA, the alert features of each stage, and transitions between stages. Secondly, optimized Baum-Welch and Viterbi algorithms are proposed, combined with the forward-backward algorithm to train the parameter of the Gaussian mixture hidden Markov model and detect the attack sequence of MSA. Finally, an improved transfer learning method is proposed, which addresses the sparsity of labeled data in MSA scenarios, a Kullback-Leibler (KL) divergence value is added as a penalty term to narrow the distribution differences between the source and target domains and solves the bias problem in the transfer learning process. The proposed algorithm is validated on the datasets DARPA 2000 and CSE-CIC-IDS2018, and the effectiveness and superiority is verified on multiple evaluation indicators. Note to Practitioners —Network attacks gradually show the large-scale, coordinated and multi-stage characteristics. Complex multi-step attacks with strong concealment and persistence have become the development trend of network attacks, which seriously threaten and infringe the secure storage and transmission of information. Most existing studies use hidden Markov model (HMM) to model multi-stage network attacks. HMM is usually more suitable for multi-step attacks occurring in a specific sequence within a continuous time interval. However, in actual multi-stage network attacks, attackers do not need to follow the exact sequence of multi-step attacks, and the intervals between successive stages of an attack can be hours, days, or even months. Attackers may also perform interleaved attacks to hide attacks. Therefore, this paper proposes a multi-stage network attack detection algorithm based on Gaussian hybrid hidden Markov and transfer learning. The optimized Gaussian hybrid hidden Markov model is used to model the alert data of multi-stage network attacks, and the improved transfer learning method is adopted to apply the knowledge learned from the source domain to the multi-stage network attack detection model of the target domain. The experimental results show that the proposed algorithm can effectively process the alert data of different attack stages under complex multi-stage network attacks, distinguish the real threat alert, false alert and irrelevant alert, and improve the performance of detecting multi-stage network attacks. The method presented in this paper can provide a valuable solution for complex multi-stage network attack detection such as advanced persistent threat (APT). Future work will further combine adversarial generation network methods to avoid the interference of adversarial attack samples, and explore more ways to improve the performance of multi-step attack detection.