Efficient malware detection using hybrid approach of transfer learning and generative adversarial examples with image representation

Abstract Identifying malicious intent within a program, also known as malware, is a critical security task. Many detection systems remain ineffective due to the persistent emergence of zero‐day variants, despite the pervasive use of antivirus tools for malware detection. The application of generative AI in the realm of malware visualization, particularly when binaries are depicted as colour visuals, represents a significant advancement over traditional machine‐learning approaches. Generative AI generates various samples, minimizing the need for specialized knowledge and time‐consuming analysis, hence boosting zero‐day attack detection and mitigation. This paper introduces the Deep Convolutional Generative Adversarial Network for Zero‐Shot Learning (DCGAN‐ZSL), leveraging transfer learning and generative adversarial examples for efficient malware classification. First, a normalization method is proposed, resizing malicious images to 128 × 128 or 300 × 300 for standardized input, enhancing feature transformation for improved malware pattern recognition. Second, greyscale representations are converted into colour images to augment feature extraction, providing a richer input for enhanced model performance in malware classification. Third, a novel DCGAN with progressive training improves model stability, mode collapse, and image quality, thus advancing generative model training. We apply the Attention ResNet‐based transfer learning method to extract texture features from generated samples, which increases security evaluation performance. Finally, the ZSL for zero‐day malware presents a novel method for identifying previously unknown threats, indicating a significant advancement in cybersecurity. The proposed approach is evaluated using two standard datasets, namely dumpware and malimg, achieving malware classification accuracies of 96.21% and 98.91%, respectively.