Automated Mobile UI Test Fragility: An Exploratory Assessment Study on Android

Automated UI testing suffers from fragility due to contin- uous – although minor – changes in the UI of applications. Such fragility has been shown especially for the web domain, though no clear evidence is available for mobile applications. Our goal is to perform an exploratory assessment of the extent and causes of the fragiliy of UI automated tests for mobile applications. For this purpose, we analyzed a small test suite -that we developed using five different testing frameworks- for an Android application (K-9 Mail) and observed the changes induced in the tests by the evolution of the UI. We found that up to 75% of code-based tests, and up to 100% of image recognition tests, had to be adapted because of the changes induced by the evolution of the application between two different versions. In addition we identified the main causes of such fragility: changes of identifiers, text or graphics, removal or relocation of elements, activity flow variation, execution time variation, and usage of physical buttons. The preliminary assessment showed that the fragility of UI tests can be a relevant issue also for mobile applications. A few common causes were found that can be used as the basis for providing guidelines for fragility avoidance and repair.