MalFSCIL: A Few-Shot Class-Incremental Learning Approach for Malware Detection

The continuous evolution of malware is posing a serious threat to personal privacy, enterprise data security, and global network infrastructure. For example, attackers can use phishing emails, botnets, etc. to induce victims to execute malware for nefarious purposes such as stealing sensitive information. Therefore, it is significant to develop effective and efficient methods to detect malware. Towards this, most state-of-the-art methods are focused on learning-based method. In order to adapt to the characteristics of sample scarcity and dynamic evolution of malware detection tasks, few-shot class incremental learning has been proposed as an efficient pairwise solution. Nevertheless, they still face two major challenges: 1) Catastrophic Forgetting: the erosion of existing knowledge by newly acquired knowledge during incremental learning. 2) Decision boundary confusion: after continuous multiple incremental sessions, the discriminative ability of the classification model is weakened. To address the above challenges, we propose a new Malware detection framework based on Few-Shot Class Incremental Learning, MalFSCIL, which utilizes a decoupled training strategy combined with a variational autocoder to mitigate catastrophic forgetting, and designs a dynamic boundary delineation method based on class prototyping to achieve accurate delineation of incremental decision boundaries. Extensive experimental results show that the proposed method outperforms the state-of-the-art techniques in malware detection and classification with high classification accuracy with open-source dataset and Internal enterprise dataset.