Flow Interaction Graph Analysis: Unknown Encrypted Malicious Traffic Detection

Nowadays traffic on the Internet has been widely encrypted to protect its confidentiality and privacy. However, traffic encryption is always abused by attackers to conceal their malicious behaviors. Since encrypted malicious traffic is similar to benign flows, it can easily evade traditional detection. In particular, the existing encrypted traffic detection methods are supervised which rely on the prior knowledge of known attacks (e.g., labeled datasets). Detecting unknown encrypted malicious traffic, which does not require prior knowledge, is still an open problem. In this paper, we propose, an unsupervised machine learning (ML) based malicious traffic detection system. Particularly, is able to detect unknown patterns of encrypted malicious traffic by utilizing a graph built upon flow interaction patterns, instead of learning the features of specific known attacks. We develop an unsupervised graph learning method to detect abnormal interaction patterns by analyzing the graph features, which allows to detect unknown attacks without requiring any labeled datasets. Moreover, we establish an information theory model to prove the effectiveness of . We show the performance of by real-world experiments with 140 attacks. The experimental results illustrate that outperforms the state-of-the-art methods by 13.9% accuracy improvement. Moreover, achieves 15.82 Mpps detection throughput with the average detection latency of 0.29s.