Selecting the Best Reliability Model to Predict Residual Defects in Open Source Software

Predicting residual defects (i.e. remaining defects or failures) in Open Source Software (OSS) may help in decision making about their adoption. Several methods exist for predicting residual defects in software. A widely used method is Software reliability growth models (SRGMs). SRGMs have underlying assumptions, which are often violated in practice, but empirical evidence has shown that many models are quite robust despite these assumption violations. However, within the SRGM family, many models are available, and it is often difficult to know which models are better to apply in a given context. We present an empirical method that applies various SRGMs iteratively on OSS defect data and selects the model which best predicts the residual defects of the OSS. The inputs of the SRGMs are the cumulative defect data grouped by weeks and the output is the number of estimated residual defects in the software. This value is a key factor for decision making about adoption of the OSS. We validate empirically the method applying it to defect data collected from twenty-one different releases of seven OSS projects. The method selects the best model 17 times out of 21. In the remaining four it selects the second best model.