Machine Learning Based Network Traffic Classification with Cosine-Similarity Based Out-of-Distribution Detection

The increasing volume of encrypted network traffic is growing at an alarming rate. This development has created intricate problems in traffic classification which is vital for effective cybersecurity. Moreover, most frameworks seem to ignore OOD detection, model calibration and novel pattern detection as cornerstone problem areas. This paper presents a machine learning approach aimed at resolving encrypted traffic classification issues and focuses on novel Out-Of-Distribution (OOD) detection and calibration issues. Primary contributions comprise detection of OOD states using softmax scaled cosine similarity, advanced variance-based feature elimination, and lowering ECE using stringent NNs. we developed a novel framework which achieves reliable and high-accurate performance across different classes of traffic. The results underscore the impact of the proposed solutions on improving dynamics, reliability, and interpretability of machine learning models and through them secure and intelligent monitoring of network traffic.