Adjusted Trust Score: A Novel Approach for Estimating the Trustworthiness of Software Defect Prediction Models

Software defect prediction (SDP) techniques play a crucial role in identifying defective code regions and improving testing efficiency. Over recent decades, a plethora of SDP approaches has emerged, with machine learning (ML) models being the most widely employed. Despite their superior predictive performance, their black-box nature and uncertainties make it challenging for developers to trust their predictions. To address this issue, we propose a novel trustworthiness score, the adjusted trust score (ATS), which helps determine when to rely on classifier predictions. Furthermore, we employ ATS to develop a reject option for SDP models. Comprehensive experiments on 32 benchmark datasets and six prevalent ML classifiers reveal that high (low) ATS values successfully yield high precision in identifying correct (or incorrect) predictions. ATS also demonstrates superiority over its counterparts, as evidenced by the Wilcoxon signed-rank test. Furthermore, a comparative analysis of prediction performance, with and without a reject option, confirms the feasibility of designing a reject option for SDP models utilizing ATS. Our work highlights that ATS can assist developers in better comprehending the strengths and weaknesses of SDP models. Therefore, it is an essential component for guaranteeing trust from developers and deserves further investigation.