A K-Means clustering and SVM based hybrid concept drift detection technique for network anomaly detection

• Using K-Means clustering to reduce the sample size of captured network traffic. • Development of two drift detection techniques for handling drift. • Measure severity of concept drift. Today’s internet data primarily consists of streamed data from various applications like sensor networks, banking data and telecommunication data networks. A new field of study, data stream mining has been gaining popularity to study streamed data behavior. Detection of anomalies in the network traffic also finds its applicability in this context. However traditional machine learning algorithms suffer in providing consistent high accuracy values and give high false alarms. This is due to the presence of concept drift in the captured data streams. Concept drift describes unknown changes in the characteristics of network data over time. Therefore, to handle presence concept drift new methodologies and techniques for drift detection, understanding and adaptation are required. In this paper, we have proposed two techniques, an Error Rate Based Concept Drift Detection and Data Distribution Based Concept Drift Detection and studied their impact. Furthermore, sliding window based data capturing and drift analyzing combined with K-Means Clustering has been used for reducing data size and upgrading training dataset. We have used the Support Vector Machine (SVM) classifier for anomaly detection and retraining of the model has been initiated based on statistical tests. The experiments have been performed on three datasets, namely, generated Testbed Dataset, NSL-KDD and CIDDS-2017. Detection accuracy, KL-Divergence and Kappa Statistics have been used to study the severity of the concept drift in the datasets. After applying the proposed approach, the SVM has been shown to have a better classification accuracy of 93.52%, 99.80% and 91.33% respectively. We achieved a precision rate of 91.84%, 99.1% and 88.3%, a recall rate of 94.30%, 99.2% and 91.7% with an F1 score of 92.9%, 99.15% and 89.6% respectively.