Deep Learning-Based Vulnerable Function Detection: A Benchmark

The application of Deep Learning (DL) technique for code analysis enables the rich and latent patterns within software code to be revealed, facilitating various downstream tasks such as the software defect and vulnerability detection. Many DL architectures have been applied for identifying vulnerable code segments in recent literature. However, the proposed studies were evaluated on self-constructed/-collected datasets. There is a lack of unified performance criteria, acting as a baseline for measuring the effectiveness of the proposed DL-based approaches. This paper proposes a benchmarking framework for building and testing DL-based vulnerability detectors, providing six built-in mainstream neural network models with three embedding solutions available for selection. The framework also offers easy-to-use APIs for integration of new network models and embedding methods. In addition, we constructed a real-world vulnerability ground truth dataset containing manually labelled 1,471 vulnerable functions and 1,320 vulnerable files from nine open-source software projects. With the proposed framework and the ground truth dataset, researchers can conveniently establish a vulnerability detection baseline system for comparison and evaluation. This paper also includes usage examples of the proposed framework, aiming to investigate the performance behaviours of mainstream neural network models and providing a reference for DL-based vulnerability detection at function-level.