Intrusive Test Automation with Failed Test Case Clustering

Regression testing is an indispensable process in software development, which ensures stable features have not been adversely broken by new changes. When GUI plays an important role in an application, a popular choice to automate the tests is applying GUI Capture/Replay tools. Unfortunately, in many applications which render images as output, the correctness of a replay run can no longer be straightforwardly verified. In this paper, we propose a test automation solution, called Intrusive Test Automation, which uses program instrumentation technique to collect the runtime internal information of a program. As a result, the correctness of a test run can be verified by the runtime traces. In addition, when large number of failed test cases are reported by the test automation system, recommending some representative test cases as a start for debugging can be helpful to programmers. This paper proposes a clustering technique based on the information collected from the instrumented code. In principle, fixing bugs in one representative test case can fix its related failed test cases as well. A case study is presented to demonstrate the effectiveness of the approach.