WHPar: a novel API parameter handling method for malware detection

Abstract In dynamic malware detection, analyzing application programming interface (API) parameters is proven to effectively complement the security information provided by APIs. Previous research on API parameters has focused on the security behavior of APIs and their parameters, while overlooking the behavioral correlation among parameters. This oversight results in limited generalizability of detection models and diminished accuracy. In this study, we present WHPar, a novel deep neural network-based malware detection approach for analyzing API parameters to identify behavioral relationships among them. It first employs the Word2Vec method to capture context-containing Information on security behavior from the API and its parameters, respectively. Then, it employs discrete cosine transform from the perceptual hash algorithm to transform the contextual information and perform embedding, for a sequence containing more comprehensive behavioral information derived from the API parameters. Finally, it feeds the sequences into the bidirectional long short-term memory model for training a binary classifier to detect malware. Experimental results demonstrate that WHPar significantly outperforms baseline methods. Moreover, when malicious samples are less prevalent than benign samples, WHPar yields superior detection results compared with other established methods in this field.