WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing
WebTestBench evaluates end-to-end web testing with models scoring below 30% F1, highlighting significant performance gaps.
Key Findings
Methodology
Using WebTestBench, diverse web applications across 7 categories were synthesized, with 100 samples. The process decomposed testing into Checklist generation and defect detection, employing the WebTester framework. Models like GPT-5.1 and Claude were evaluated on metrics including coverage, F1, precision, and recall. The evaluation involved semantic matching, logical reasoning, and simulation of human interactions via Playwright MCP, providing a comprehensive assessment of models’ capabilities in complex, open-ended web environments.
Key Results
- All evaluated models scored below 30% F1, with GPT-5.1 reaching 26.4%, primarily due to a recall of 33.3%. Checklist coverage averaged around 65%, indicating many test cases were omitted, leading to incomplete defect detection. Precision was approximately 30%, with high false-positive rates, reflecting models’ difficulty in accurately identifying subtle defects. Performance varied across web categories, especially in content-rich and logically complex scenarios, exposing the gap between current AI capabilities and industrial demands.
- In the Checklist generation phase, coverage was insufficient, missing critical test items, which compromised defect detection. During defect detection, models struggled with false positives and negatives, especially in logical constraint scenarios such as overlapping bookings or form validation. The results underscore the need for enhanced reasoning and interaction understanding in models to meet real-world testing requirements.
- Cross-category analysis revealed that models perform better on simpler, static pages but falter with dynamic, content-rich, or logically constrained web applications. The experiments demonstrated that integrating semantic matching and logical inference mechanisms can improve detection accuracy, but overall, current models are not yet ready for deployment in high-stakes industrial testing environments.
Significance
This research establishes a systematic benchmark for evaluating AI-driven web testing, addressing the limitations of static visual similarity and predefined checklists. By incorporating logical constraints and long-term interactions, it pushes the frontier of automated testing, providing critical insights into the current state and future needs of AI in software quality assurance. The findings highlight the substantial gap between research progress and industrial deployment, guiding future development towards more robust, reasoning-capable models. Ultimately, this work accelerates the integration of AI into software engineering workflows, promising more reliable, efficient web development and testing processes.
Technical Contribution
The paper introduces WebTestBench, a comprehensive benchmark covering multiple web categories with detailed annotations and evaluation metrics. It proposes WebTester, a two-stage framework combining semantic checklist generation and defect detection through simulated interactions and logical reasoning. The evaluation protocol includes semantic matching via models like Qwen3.5-27B, and logical constraint verification, setting a new standard for end-to-end web testing. The experimental results reveal critical performance bottlenecks, emphasizing the need for models with enhanced reasoning and interaction capabilities. This work bridges the gap between static testing methods and dynamic, logic-aware evaluation, offering a foundation for future AI advancements in web testing automation.
Novelty
This is the first benchmark to systematically evaluate large language models on end-to-end web testing, explicitly incorporating latent logical constraints and long-horizon interactions. Unlike prior works focusing solely on static visual similarity or predefined scripts, WebTestBench emphasizes the model’s ability to understand complex, dynamic web environments. The integration of semantic matching with logical reasoning mechanisms represents a significant innovation, providing a more realistic and comprehensive assessment of AI capabilities in software testing. This approach sets a new direction for research in automated web QA, emphasizing reasoning, interaction, and logical consistency.
Limitations
- Models exhibit limited understanding of complex web interactions, especially in logical constraints, leading to high false negatives and positives. The evaluation environment relies on simulated interactions, which may not fully capture real user behaviors, limiting ecological validity. Computational costs are high, and inference speed remains a challenge for industrial deployment. Future work should focus on improving reasoning abilities, reducing costs, and expanding scenario diversity to enhance model robustness and applicability.
Future Work
未来将结合强化学习和多模态信息融合技术，增强模型在长远交互和逻辑推理中的表现。计划扩展样本库，丰富网页场景，提升模型泛化能力。同时，探索多模态数据（如视觉、声音）融合，增强模型对复杂网页内容的理解。还将优化推理速度和成本，推动模型在工业环境中的应用落地，最终实现更智能、更可靠的自动网页测试体系。
AI Executive Summary
WebTestBench代表了自动化网页测试的重大突破。随着大语言模型（如GPT-5.1、Claude）在网页生成和代码编写中的应用日益成熟，用户可以用自然语言快速创建网页，但随之而来的质量验证问题变得尤为关键。传统的静态视觉比对和手工预定义测试用例在复杂交互和潜在逻辑约束方面表现不足，难以满足工业级的需求。为此，本文提出WebTestBench，涵盖7个网页类别、100个样本，模拟端到端的自动化测试流程。
在此基础上，设计了WebTester框架，结合Checklist生成和缺陷检测两个环节，利用语义匹配和逻辑推理机制进行性能评估。实验结果显示，所有模型的F1值均低于30%，表现出明显的性能瓶颈。测试覆盖率不足70%，漏检和误判问题严重，特别是在内容丰富、交互复杂的网页环境中。模型在逻辑约束检测方面尤为薄弱，未能有效识别潜在的逻辑错误。这些发现揭示了当前大模型在工业网页自动化测试中的不足，强调了模型在交互理解和逻辑推理方面的提升空间。
研究不仅为模型优化提供了具体指标，也为自动化测试技术的理论发展奠定了基础。未来，WebTestBench将推动AI在软件质量保障中的深度应用，加速软件工程自动化，具有重要的学术和工业意义。
Deep Dive
Abstract
The emergence of Large Language Models (LLMs) has catalyzed a paradigm shift in programming, giving rise to "vibe coding", where users can build complete projects and even control computers using natural language instructions. This paradigm has driven automated webpage development, but it introduces a new requirement about how to automatically verify whether the web functionalities are reliably implemented. Existing works struggle to adapt, relying on static visual similarity or predefined checklists that constrain their utility in open-ended environments. Furthermore, they overlook a vital aspect of software quality, namely latent logical constraints. To address these gaps, we introduce WebTestBench, a benchmark for evaluating end-to-end automated web testing. WebTestBench encompasses comprehensive dimensions across diverse web application categories. We decompose the testing process into two cascaded sub-tasks, checklist generation and defect detection, and propose WebTester, a baseline framework for this task. Evaluating popular LLMs with WebTester reveals severe challenges, including insufficient test completeness, detection bottlenecks, and long-horizon interaction unreliability. These findings expose a substantial gap between current computer-use agent capabilities and industrial-grade deployment demands. We hope that WebTestBench provides valuable insights and guidance for advancing end-to-end automated web testing. Our dataset and code are available at https://github.com/friedrichor/WebTestBench.