An Analysis of the Test Repair Capability of LLMs in Android GUI Testing

Automated GUI testing is a crucial activity in modern Android development, yet its nature is notoriously fragile, especially when the GUI is built using dynamic frameworks like Jetpack Compose. Minor UI changes frequently break tests, flooding continuous integration pipelines with false positives, and burdening developers with costly repairs. To reduce test repair effort, we evaluate a developer-in-the-loop approach leveraging a Large Language Model, GitHub Copilot with Claude 3.7 Sonnet, as a zero-shot repair agent within Android Studio. By analyzing IDE context, this method updates broken selectors, adjusts test oracles, and maintains test semantics after GUI changes. We empirically evaluate the approach using the Bitwarden mobile app for Android, an open-source project containing 1083 GUI tests. We analyzed the test suite across two recent application versions, reporting failures and using the LLM to repair the failing tests. Our evaluation investigates test fragility, effectiveness of zero-shot LLM-based repairs, additional benefits from retrying prompts, and improvements from brief developer interactions. Results show that a single zero-shot prompt recovers a significant proportion of failing tests, reducing manual maintenance efforts. Another retry provided minimal additional benefits, whereas brief developer interactions considerably enhanced recovery rates. Our findings indicate that integrating LLM-driven techniques substantially eases the maintenance burden of GUI test suites, ensuring robustness against rapid UI evolution in Jetpack Compose applications.