DPRFuzz: Enhancing Vulnerability Mining With Two-Stage Reinforcement Learning

American fuzzy lop (AFL), as a representative tool for fuzzing, is capable of uncovering security vulnerabilities in industrial systems. It suffers from consuming a large amount of computational resources during the mutation. To improve the performance of AFL, researchers adopt algorithms, such as particle swarm optimization and long short-term memory, to optimize mutation operator selection. However, challenges persist in these approaches integrated with AFL, including optimization model complexity, insufficient accuracy, and poor generalization scalability. To address these issues, the article proposes a new fuzzer called DPRFuzz to optimize AFL’s mutation phases. First, in the deterministic mutation strategy mutation phase, deep Q network and trust region policy optimization are leveraged to precisely generate effective mutated samples through perceiving mutation process in a relatively short time. Then, to boost the efficiency of the Havoc random mutation phase, we improve the Thompson sampling algorithm based on a multiagent strategy to generate an overall optimal mutation strategy chain. Finally, the approach is tested on eight programs, such as readelf, tcpdump, and nm, and the advantages of DPRFuzz are analyzed. Most importantly, the experiment reveals results that DPRFuzz achieves better fuzzing performance compared to the traditional and other AFL-based fuzzers, such as AFL, AFL++, AFLSmart, etc. On average, DPRFuzz achieves an improvement in code coverage of over 10%, along with a significant increase in the number of crashes.