Mutation Analysis of Magento for Evaluating Threat Model-Based Security Testing

Security testing is a major means for assuring software security and many security testing techniques have been developed in the past. Benchmarks, however, are in great demands for empirically evaluating the vulnerability detection capabilities of these techniques. To develop such a benchmark, this paper presents an approach to security mutation analysis of Magento, a fully-fledged open source e-commerce web application for evaluating automated security testing techniques. We create security mutants by injecting vulnerabilities in a systematic way. Specifically, we consider the causes of vulnerabilities according to OWASP's top 10 web application security risks, the application's business logic, as well as various consequences of vulnerabilities (i.e., STRIDE attacks). We have created 63 mutants and applied them successfully to the evaluation of two security testing techniques that use threat trees and threat nets as threat models for test generation. Our experiments show that these testing methods can kill most of the mutants but cannot detect the vulnerabilities that are not captured by the threat models.