Estimating Fault Numbers Remaining After Testing

Testing is an essential component of the software development process, but also one which is exceptionally difficult to manage and control. For example, it is well understood that testing techniques are not guaranteed to detect all faults, but more frustrating is that after the application of a testing technique the tester has little or no knowledge of how many faults might still be left undiscovered. This paper investigates the performance of a range of capture-recapture models to determine the accuracy with which they predict the number of defects remaining after testing. The models are evaluated with data from two empirical testing-related studies and from one larger publicly available project and the factors affecting the accuracy of the models are analysed. The paper also considers how additional information (such as structural coverage data) may be used to improve the accuracy of the estimates. The results demonstrate that diverse sets of faults resulting from different testers using different techniques tend to produce the most accurate results, and also illustrate the sensitivity of the estimators to the patterns of fault data.