AI in GUI-based testing: A survey of techniques, tools, and perceived advantages and limitations

Background: The adoption of Artificial Intelligence (AI) techniques in Software Testing (ST) has grown rapidly, particularly in response to the increasing complexity of modern systems. In GUI-based testing, AI is often cited as a promising means to automate repetitive tasks and improve testing efficiency. However, the actual use of AI in this domain remains underexplored through systematic empirical investigation. Objective: This study aims to analyze how AI is adopted in GUI-based testing, identifying the techniques and tools employed, the testing activities they support, and the perceived benefits and limitations. Method: We conducted a large-scale survey involving 107 participants from both academia and industry. The survey focuses on three core testing activities: test case definition, test oracle design, and test case optimization. It extends a prior study based on interviews with 45 industry practitioners. Results:Findings show that AI is primarily used to support test case definition, with techniques such as Natural Language Processing, Optimization, and Large Language Models (LLMs) being the most common. AI also provides support in test oracle design, where image processing and knowledge representation play key roles, and in test suite optimization, through the use of supervised learning, reinforcement learning, and search-based techniques. Conclusion: The paper identifies ongoing challenges and outlines future directions, including the need for transparent AI tools, guidelines for LLM integration, and the deployment of a continuously open survey to monitor trends in AI adoption over time.