Maintenance of Android Widget-based GUI Testing: A Taxonomy of test case modification reasons

Automated GUI testing is a fundamental part of the V&V process of every software, but -especially for fastly evolving applications- is linked to serious maintenance costs. Android (and mobile, in general) apps are also particularly prone to fragility, which means that even minor modifications in the graphical user interface and in the way it is interacted may create the need for relevant adaptation effort on test code. Based on a preliminary exploration about the evolution and fragility of test suites that we made on a set of nearly 20 thousand Android open-source applications hosted on GitHub, we performed a deeper inspection about the causes underlying the modifications needed in test classes and methods. To do so, we manually examined a set of 423 projects featuring the Espresso automated GUI testing tool, and 945 diff files containing modifications in test methods. We derived a set of 27 different causes of modifications, that we subdivided under nine different macro-categories, two related to modifications performed just in code and application logic, and seven pertaining modifications in the GUI functioning and appearance, and hence related to our definition of GUI testing fragility. Based on the set of projects we inspected, we conclude that modifications related to changes in the GUI occured in 55% of modified test methods, a higher frequency than the ones related to changes in test logic only (35.76%) and to adaptations to modified application logic (27.19%). The described taxonomy can serve as a basis for further research, aimed at giving guidelines to developers to avoid fragility, and reduce the maintenance costs for automated test suites for Android applications.