A benchmarking-inspired approach to determine threshold values for metrics

In order to assess software quality by software metrics, usually, thresholds for metric values are needed. A common problem is to define reasonable threshold values. One possible solution is to use a benchmarking approach: the threshold value for a metric is calculated based on the metric values of a set of systems, which are called benchmarking base. A relevant question is, how the used benchmarking base inuences the result of the software quality assessment. Based on the quality assessment approach of Quamoco, we conduct a series of experiments, using different benchmarking bases. For each benchmarking base a quality assessment of a series of test systems is conducted. We analyze the whether the quality assessment results of the test systems are concordant for different benchmarking bases. The main findings are: (1) The bigger the benchmarking base, the less divergent are the rankings, and the less is the variance of the results. (2) The size of the systems contained within a benchmarking base does not inuence the results, i.e. a benchmarking base containing small systems works equally well for small and large systems, and vice versa. These results show that benchmarking is a feasible approach for determining threshold values.