WebForge: Breaking the Realism-Reproducibility-Scalability Trilemma in Browser Agent Benchmark
WebForge automates the end-to-end creation of realistic, reproducible, and scalable web environments using a four-stage pipeline, enabling multi-dimensional capability profiling.
Key Findings
Methodology
WebForge employs a four-stage pipeline: Plan, Generate, Refine, and Validate. The Plan module uses dual large language models (high-temperature for creative task drafting, low-temperature for logical verification) to design structured tasks with a seven-dimensional difficulty vector. The Generate module constructs fully functional web environments with real data, embedding anti-cheating mechanisms. The Refine stage injects real-web noise—pop-ups, delays, cookie prompts—to enhance realism. Validation executes tasks within a Chromium browser, replaying solution paths to verify solvability, filtering out defective tasks. The framework supports fine-grained difficulty regulation across navigation, visual complexity, reasoning, etc., enabling detailed capability analysis. The final benchmark, WebForge-Bench, contains 934 tasks across 7 domains and 3 difficulty levels, facilitating multi-model capability assessment.
Key Results
- Experiments demonstrate that difficulty stratification effectively distinguishes model capabilities; success rates decline from 75.9% at L1 to 58.0% at L3, confirming the effectiveness of the seven-dimensional difficulty control.
- Cross-domain analysis reveals capability biases; models perform best in info retrieval and content creation, but struggle with consumer transactions and moderation, highlighting the importance of multi-dimensional evaluation.
- Ablation studies show that each pipeline component—planning, environment generation, noise injection, validation—is essential for environment quality and task solvability, underscoring the integrated design's robustness.
Significance
This work addresses the longstanding challenge of creating web benchmarks that are simultaneously realistic, reproducible, and scalable. By automating environment generation with multi-dimensional difficulty control, it overcomes content drift, manual curation costs, and environment oversimplification. The framework enables comprehensive capability profiling, revealing strengths and biases across different task types and environmental complexities. It advances the field by providing a standardized, scalable, and detailed evaluation platform, fostering progress in autonomous web agents and related AI systems. Its multi-faceted approach aligns with industry needs for reliable, large-scale testing of intelligent web interactions.
Technical Contribution
WebForge introduces a novel end-to-end automation pipeline that integrates dual-model task design, real-web noise injection, and in-browser validation. The seven-dimensional difficulty framework allows precise, multi-aspect task regulation, surpassing prior static or single-dimension approaches. The system’s ability to generate fully interactive, realistic web environments without manual annotation is a significant engineering breakthrough. It also establishes a new standard for multi-dimensional capability assessment, combining environment realism with scalable automation, thus enabling detailed, nuanced performance analysis of diverse models.
Novelty
This is the first comprehensive system to automate the creation of interactive, multi-dimensional web benchmarks that incorporate real data, web noise, and in-browser validation. Unlike prior static or limited-dimension methods, WebForge’s integrated pipeline ensures environments are both realistic and scalable, with fine-grained difficulty control. Its use of dual large language models for task blueprinting and environment injection of real-web noise distinguishes it from existing approaches like DyVal or TaskBench, which lack multi-dimensional difficulty regulation and environment realism.
Limitations
- The environment generation relies on predefined templates and real data sources, which may limit coverage of highly specialized or novel web interactions.
- While multi-model validation improves robustness, some complex or edge-case tasks may still slip through, especially under extreme difficulty settings.
- Injecting real-web noise introduces stochastic variability, potentially affecting reproducibility across different runs or models.
Future Work
Future directions include enhancing environment diversity via dynamic data sourcing, refining difficulty regulation algorithms with reinforcement learning, and extending multi-modal capabilities. Developing adaptive noise injection techniques to balance realism and stability, and exploring real-time environment adjustment based on model performance, will further improve evaluation robustness. Additionally, integrating user behavior modeling and multi-agent interactions could broaden the benchmark’s applicability for complex web scenarios.
AI Executive Summary
WebForge represents a significant advancement in the automatic construction of web benchmarks for evaluating autonomous agents. Addressing the core trilemma of realism, reproducibility, and scalability, it employs a four-stage pipeline—Plan, Generate, Refine, and Validate—to produce fully interactive, self-contained web environments. The key innovation lies in leveraging dual large language models to design structured tasks with a seven-dimensional difficulty vector, ensuring fine-grained control over complexity. The environment construction process incorporates real web data and noise injection, mimicking authentic browsing conditions, while the validation stage executes tasks within a real browser to verify solvability, filtering out defective environments.
The framework’s multi-dimensional difficulty control enables detailed capability profiling, revealing strengths and biases across navigation, visual complexity, reasoning, and other axes. The authors built WebForge-Bench, comprising 934 tasks across 7 domains and 3 difficulty levels, facilitating comprehensive multi-model evaluation. Experimental results demonstrate that difficulty stratification effectively distinguishes model performance, with success rates decreasing systematically as difficulty increases. Cross-domain analysis uncovers capability biases invisible to aggregate metrics, emphasizing the importance of multi-faceted evaluation.
This work addresses critical limitations of existing benchmarks, such as content drift, environment oversimplification, and manual curation costs. By automating environment generation and incorporating realistic web noise, WebForge offers a scalable, reliable platform for advancing AI capabilities in web navigation and interaction tasks. Its detailed, multi-dimensional assessment framework paves the way for more nuanced understanding of model strengths and weaknesses, guiding future research and industry applications. Future efforts will focus on expanding environment diversity, refining difficulty regulation, and integrating multi-modal data, aiming to create even more robust and adaptable benchmarking tools for AI-driven web agents.
Deep Dive
Abstract
Existing browser agent benchmarks face a fundamental trilemma: real-website benchmarks lack reproducibility due to content drift, controlled environments sacrifice realism by omitting real-web noise, and both require costly manual curation that limits scalability. We present WebForge, the first fully automated framework that resolves this trilemma through a four-agent pipeline -- Plan, Generate, Refine, and Validate -- that produces interactive, self-contained web environments end-to-end without human annotation. A seven-dimensional difficulty control framework structures task design along navigation depth, visual complexity, reasoning difficulty, and more, enabling systematic capability profiling beyond single aggregate scores. Using WebForge, we construct WebForge-Bench, a benchmark of 934 tasks spanning 7 domains and 3 difficulty levels. Multi-model experiments show that difficulty stratification effectively differentiates model capabilities, while cross-domain analysis exposes capability biases invisible to aggregate metrics. Together, these results confirm that multi-dimensional evaluation reveals distinct capability profiles that a single aggregate score cannot capture. Code and benchmark are publicly available at https://github.com/yuandaxia2001/WebForge.