Background. Evidence from empirical studies suggests that mobile applications are not thoroughly tested as their desktop counterparts. In particular, GUI testing is generally limited. Like web-based applications, mobile apps suffer from GUI testing fragility, i.e., GUI test classes failing or needing interventions because of modications in the AUT or in its GUI arrangement and definition. Aims. The objective of our study is to examine the diffusion of test classes created with a set of popular GUI Automation Frameworks for Android apps, the amount of changes required to keep test classes up to date, and the amount of code churn in existing test suites, along with the underlying modications in the AUT that caused such modications. We defined 12 metrics to characterize the evolution of test classes and test methods, and a taxonomy of 28 possible causes for changes to test code. Method. To perform our experiments, we selected six widely used open-source GUI Automation Frameworks for Android apps. We evaluated the diffusion of the tools by mining the GitHub repositories featuring them, and computed our set of metrics on the projects. Applying the Grounded Theory technique, we then manually analyzed diff files of test classes written with the selected tools, to build from the bottom up a taxonomy of causes for modications of test code. Results. We found that none of the considered GUI automation frameworks achieved a major diffusion among open-source Android projects available on GitHub. For projects featuring tests created with the selected frameworks, we found that test suites had to be modied often { specifically, about 8% of developers' modified LOCs belonged to test code and that a relevant portion (around 50% on average) of those modications were induced by modications in GUI definition and arrangement. Conclusions. Test code written with GUI automation frameworks proved to need significant interventions during the lifespan of a typical Android open-source project. This can be seen as an obstacle for developers to adopt this kind of test automation. The evaluations and measurements of the maintenance needed by test code written with GUI automation frameworks, and the taxonomy of modication causes, can serve as a benchmark for developers, and the basis for the formulation of actionable guidelines and the development of automated tools to help mitigating the issue.

Scripted GUI Testing of Android Open-Source Apps: Evolution of Test Code and Fragility Causes / Coppola, Riccardo; Morisio, Maurizio; Torchiano, Marco; Ardito, Luca. - In: EMPIRICAL SOFTWARE ENGINEERING. - ISSN 1382-3256. - ELETTRONICO. - 24:5(2019), pp. 3205-3248. [10.1007/s10664-019-09722-9]

Scripted GUI Testing of Android Open-Source Apps: Evolution of Test Code and Fragility Causes

Riccardo Coppola;Maurizio Morisio;Marco Torchiano;Luca Ardito
2019

Abstract

Background. Evidence from empirical studies suggests that mobile applications are not thoroughly tested as their desktop counterparts. In particular, GUI testing is generally limited. Like web-based applications, mobile apps suffer from GUI testing fragility, i.e., GUI test classes failing or needing interventions because of modications in the AUT or in its GUI arrangement and definition. Aims. The objective of our study is to examine the diffusion of test classes created with a set of popular GUI Automation Frameworks for Android apps, the amount of changes required to keep test classes up to date, and the amount of code churn in existing test suites, along with the underlying modications in the AUT that caused such modications. We defined 12 metrics to characterize the evolution of test classes and test methods, and a taxonomy of 28 possible causes for changes to test code. Method. To perform our experiments, we selected six widely used open-source GUI Automation Frameworks for Android apps. We evaluated the diffusion of the tools by mining the GitHub repositories featuring them, and computed our set of metrics on the projects. Applying the Grounded Theory technique, we then manually analyzed diff files of test classes written with the selected tools, to build from the bottom up a taxonomy of causes for modications of test code. Results. We found that none of the considered GUI automation frameworks achieved a major diffusion among open-source Android projects available on GitHub. For projects featuring tests created with the selected frameworks, we found that test suites had to be modied often { specifically, about 8% of developers' modified LOCs belonged to test code and that a relevant portion (around 50% on average) of those modications were induced by modications in GUI definition and arrangement. Conclusions. Test code written with GUI automation frameworks proved to need significant interventions during the lifespan of a typical Android open-source project. This can be seen as an obstacle for developers to adopt this kind of test automation. The evaluations and measurements of the maintenance needed by test code written with GUI automation frameworks, and the taxonomy of modication causes, can serve as a benchmark for developers, and the basis for the formulation of actionable guidelines and the development of automated tools to help mitigating the issue.
File in questo prodotto:
File Dimensione Formato  
emse_paper.pdf

accesso aperto

Tipologia: 2. Post-print / Author's Accepted Manuscript
Licenza: PUBBLICO - Tutti i diritti riservati
Dimensione 808.7 kB
Formato Adobe PDF
808.7 kB Adobe PDF Visualizza/Apri
Coppola2019_Article_ScriptedGUITestingOfAndroidOpe.pdf

non disponibili

Tipologia: 2a Post-print versione editoriale / Version of Record
Licenza: Non Pubblico - Accesso privato/ristretto
Dimensione 2.62 MB
Formato Adobe PDF
2.62 MB Adobe PDF   Visualizza/Apri   Richiedi una copia
Pubblicazioni consigliate

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11583/2732445