Estimating Software Vulnerabilities: A Case Study Based on the Misclassification of Bugs in MySQL Server

Software vulnerabilities are an important part of the modern software economy. Being able to accurately classify software defects as a vulnerability, or not, allows developers and end users to expend appropriately more effort on fixing those defects which have security implications. However, we demonstrate in this paper that the expected number of misclassified bugs (those not marked as also being vulnerabilities) may be quite high and thus human efforts to classify bug reports as vulnerabilities appears to be quite ineffective. We conducted an experiment using the MySQL bug report database to estimate the number of misclassified bugs yet to be identified as vulnerabilities. The MySQL database server versions we evaluated currently have 76 publicly reported vulnerabilities. Yet our experimental results show, with 95% confidence, that the MySQL bug database has between 499 and 587 misclassified bugs for the same software. This is an estimated increase of vulnerabilities between657% and 772% over the number currently identified and publicly reported in the National Vulnerability Database and the Open Source Vulnerability Database.