Espresso vs. Eyeautomate: An experiment for the comparison of two generations of android GUI testing

Context: Different approaches exist for automated GUI testing of Android applications, each with its peculiarities, advantages, and drawbacks. The most common are either based on the structure of the GUI or use visual recognition. Goal: In this paper, we present an empirical evaluation of two different GUI testing techniques with the use for each of a representative tool: (1) Visual GUI testing, with the use of EyeAutomate, and (2) Layout-based GUI testing, with the use of Espresso. Method: We conducted an experiment with a population of 78 graduate students. The participants of the study were asked to create the same test suite for a popular, open-source Android app (Omni-Notes) with both the tools, and to answer a survey about their preference to the one or the other, and the perceived difficulties when developing the test scripts. Results: By analyzing the outcomes of the delivered test suites (in terms of number of test scripts delivered and ratio of working ones) and the answers to the survey, we found that the participants showed similar productivity with both the tools, but the test suites developed with EyeAutomate were of higher quality (in terms of correctly working test scripts). The participants expressed a slight preference towards the EyeAutomate testing tool, reflecting a general complexity of Layout-based techniques -- represented by Espresso -- and some obstacles that may make the identification of components of the GUI quite a long and laborious task. Conclusions: The evidence we collected can provide useful hints for researchers aiming at making GUI testing techniques for mobile applications more usable and effective.