Discriminative World Models for Web Agents
Discriminative world models trained via predicted-state matching improve web agent decision-making and task success.
Key Findings
Methodology
This paper introduces a discriminative training approach for web world models, utilizing WebArena's multi-path decision points. The model generates textual state representations optimized to distinguish true next states from alternatives, using a pretrained language model (e.g., Qwen3-32B) as a judge. Unlike supervised next-state prediction, this method emphasizes state discriminability, enabling better downstream action ranking and end-to-end task performance. Experiments show a 80.8% accuracy in predicted-state matching, outperforming WebDreamer and WebWorld, and significantly improving PRM-based action ranking on WebPRMBench. The approach is validated in WebArena-Lite, demonstrating practical task success improvements.
Key Results
- Achieved 80.8% accuracy on held-out predicted-state matching, surpassing WebDreamer (74.5%) and WebWorld (70.2%), with robustness across judges like GPT-4 and Llama-3.
- In WebPRMBench, the use of predicted state representations improved PRM action ranking accuracy by approximately 10%, confirming the discriminative power of the representations.
- End-to-end evaluation on WebArena-Lite showed a significant increase in task success rate when integrating the model into test-time action selection, highlighting real-world applicability.
Significance
This work advances web automation by shifting from fixed-format supervised state prediction to discriminative state representations, addressing the core challenge of differentiating outcomes of competing actions. It enhances the generalization and robustness of web agents, paving the way for more intelligent, planning-capable systems that can operate effectively in complex, dynamic web environments. The methodology also provides a foundation for integrating model-based planning with reinforcement learning, promising future breakthroughs in autonomous web navigation.
Technical Contribution
The key technical innovation is the introduction of predicted-state matching, a training objective that emphasizes the discriminability of next states rather than reproducing fixed representations. The method leverages multi-path decision point data, constructing pairwise examples for training a textual state discriminator. This approach departs from traditional supervised prediction, enabling flexible, task-adaptive state representations. The integration of large pretrained language models as judges further enhances the efficiency and accuracy of state discrimination, offering a new paradigm for web world modeling and planning.
Novelty
This is the first work to formalize and implement a discriminative training objective for web world models, focusing on state distinguishability rather than fixed-format state generation. It innovatively constructs training data from multi-path decision points, enabling models to learn representations that are inherently more useful for downstream decision-making tasks. This approach addresses fundamental limitations of supervised next-state prediction, representing a significant step forward in web agent modeling.
Limitations
- The approach relies heavily on the availability of rich multi-path decision data, which can be costly to collect at scale. Its performance may degrade in environments with sparse or highly complex state changes.
- The method's dependence on large pretrained language models as judges introduces variability in discriminability across different judge architectures, potentially affecting robustness.
- Current experiments are primarily conducted in simulated environments like WebArena; real-world web pages with high variability and visual complexity remain a challenge for direct deployment.
Future Work
Future directions include integrating multi-modal signals such as visual and structural cues to enhance state discriminability, exploring reinforcement learning to optimize decision policies based on discriminative states, and scaling the approach to more complex, real-world web environments. Additionally, developing methods to reduce data collection costs and improve transferability across domains will be crucial for practical deployment.
AI Executive Summary
This research introduces a novel discriminative training framework for web world models, addressing a key limitation of traditional supervised next-state prediction. Conventional models often generate fixed-format representations that lack the ability to distinguish outcomes of similar actions, limiting their effectiveness in multi-path decision environments. By leveraging WebArena's rich dataset of branching decision points, the authors construct a training paradigm called predicted-state matching, where the model learns to produce textual state representations that can be reliably distinguished by a pretrained judge. This approach significantly improves the model's ability to differentiate between true and alternative states, achieving an 80.8% accuracy in the held-out benchmark, outperforming existing models like WebDreamer and WebWorld. When integrated into PRM-based action ranking, these discriminative representations boost accuracy by approximately 10%, demonstrating their practical utility. Moreover, in end-to-end web navigation tasks within WebArena-Lite, the approach leads to substantial improvements in task success rates, confirming its real-world relevance. The core innovation lies in shifting from reproducing fixed state formats to learning flexible, discriminative representations that better support planning and decision-making. This work opens new avenues for combining model-based planning with reinforcement learning, promising more robust and intelligent web agents capable of operating in complex, dynamic environments. Future research will focus on multi-modal integration, reducing data costs, and extending to real-world web scenarios, ultimately pushing the boundaries of autonomous web navigation technology.
Deep Analysis
Background
Web代理技术近年来快速发展，尤其是在自然语言指令驱动的网页交互任务中，代表性工作如WebDreamer和WebWorld采用监督学习预测下一状态，推动自动化水平提升。然而，这些方法多依赖固定格式的状态表示，难以应对复杂多变的网页环境。近年来，模型预测与规划结合的研究逐渐兴起，利用多路径决策和模型预测改善决策质量，但仍存在状态判别能力不足的问题。传统方法多关注生成静态状态，忽视了状态的判别性，限制了模型在复杂环境中的泛化能力。
Core Problem
核心问题在于，现有Web世界模型多关注生成固定格式的下一状态，忽视了状态的判别性。在多路径决策环境中，模型应能区分不同候选动作导致的状态差异，以便更有效地进行动作排序和规划。传统监督预测目标在此方面表现有限，难以满足多样化网页场景下的判别需求，限制了模型的泛化和应用效果。这导致模型在实际任务中难以准确区分不同动作的结果，从而影响整体任务成功率。
Innovation
本研究的创新点在于引入判别式状态匹配训练目标，强调状态的区分能力而非单纯复制目标状态。通过多路径决策点构建判别训练集，利用预训练语言模型实现高效状态判别，突破了固定格式限制。此方法不仅提升了状态区分能力，也增强了模型在复杂网页环境中的适应性，为Web代理的决策规划提供了新工具。与传统监督方法相比，该方法更关注状态的判别性和差异性，具有更强的泛化能力。
Methodology
- �� 构建多路径决策点：从WebArena数据中提取每个状态的多个候选动作及其对应的下一状态。
- �� 训练判别模型：输入当前状态和候选动作，生成文本状态表示，利用判别器（如Qwen3-32B）判断其是否匹配真实下一状态。
- �� 目标设计：非匹配目标，而是通过判别器区分真实与候选状态，优化模型生成判别性强的状态表示。
- �� 数据采集：利用WebArena中的决策点，生成大量状态对比样本，训练判别模型。
- �� 评估指标：在判别任务中达成80%以上准确率，验证状态判别能力；在WebPRMBench中提升动作排序性能；在WebArena-Lite中实现端到端任务成功率提升。
Experiments
采用WebArena中的多路径决策数据，构建判别训练集，训练判别模型（如Qwen3-32B）。在判别任务中评估准确率，比较WebDreamer、WebWorld等模型的判别效果。随后，将判别状态表示引入PRM模型，评估动作排序性能。最后，将模型应用于WebArena-Lite端到端任务，验证实际任务成功率提升。通过不同判别器（GPT-4、Llama-3）验证模型的鲁棒性。
Results
模型在判别任务中达80.8%的准确率，优于WebDreamer（74.5%）和WebWorld（70.2%）。在WebPRMBench中，状态表示提升动作排序准确率约10%，验证了判别信息的有效性。端到端任务中，结合模型的动作选择策略显著提高任务成功率，表明判别式状态表示在实际应用中具有巨大潜力。
Applications
该方法适用于Web自动化、智能助手、自动网页导航等场景，能显著提升动作决策的准确性和效率。未来可结合强化学习，优化连续决策策略，适应更复杂的网页环境，为企业和个人提供智能化网页操作解决方案。
Limitations & Outlook
当前模型依赖大量多路径决策数据，数据采集成本高，且在极端复杂网页或变化频繁场景下表现仍有限。模型对预训练语言模型的依赖较大，泛化能力有待验证。未来需增强模型鲁棒性，扩展到更复杂的网页场景。
Plain Language Accessible to non-experts
想象你在一家厨房做饭，菜单上有很多不同的菜肴。每次你选择一道菜，厨房会根据你的选择准备食材和步骤。传统的方法就像只记住了你做的那道菜的具体步骤，但无法区分不同菜肴的细微差别。本文提出的方法更像是让厨房学会区分每一道菜的关键特征，比如调料的用量、火候的变化，而不是只记住菜谱的固定步骤。这样，当你在厨房里选择不同的菜时，厨房能更准确地判断每个选择的结果，帮助你做出更好的决定。这种判别能力让厨房变得更智能，能更快帮你做出满意的饭菜。
ELI14 Explained like you're 14
想象你在玩一个游戏，你可以选择很多不同的动作，比如跳跃、跑步或者躲藏。每个动作会让你到达不同的场景，但你不总是知道哪个动作会带你到最好的地方。以前的游戏设计只告诉你每个动作的结果，但没有帮你区分哪个结果更好。现在，这个新方法就像是让游戏学会判断每个动作会带你到哪里，并告诉你哪个结果更酷、更有趣。它通过观察很多不同的场景，学会了区分每个动作的不同效果。这样，你在玩游戏时就能更聪明地选择动作，赢得更多胜利！这就像是给游戏装上了一个聪明的判断器，让它帮你做出最棒的决定。
Glossary
Predicted-State Matching (预测状态匹配)
一种训练目标，使模型生成的状态表示能区分不同候选动作的真实结果。技术上通过判别器判断匹配与否，强调状态的判别性。
用于训练Web世界模型，使其能区分不同动作引起的状态差异。
Process Reward Model (PRM, 过程奖励模型)
一种用于Web任务中的动作排序和选择的模型，基于状态和动作的预测结果进行偏好判定。
在本文中用来评估状态表示对动作排序的提升效果。
WebArena
一个Web环境数据集，包含多路径决策点，用于训练和评估Web代理的决策能力。
提供多路径决策点数据，支持判别式训练。
AXTree
网页结构的树状表示，描述网页的结构和内容变化。
作为传统的状态表示格式之一。
WebPRMBench
评估Web代理动作排序性能的基准测试，包含偏好标签和候选动作。
用于验证状态表示对动作排序的影响。
Open Questions Unanswered questions from this research
- 1 如何在极端复杂或动态网页环境中保持判别能力？未来模型是否能结合视觉信息实现多模态判别？
- 2 判别式训练在多任务、多场景迁移中的表现如何？
- 3 如何降低数据采集成本，提升模型泛化能力？
Applications
Immediate Applications
Web自动化优化
提升网页操作的决策准确性，减少人工干预，适用于企业自动化测试和个人智能助手。
智能网页导航
实现更智能的网页浏览和信息检索，提升用户体验和效率。
Long-term Vision
自主Web代理
开发具有自主学习和规划能力的Web智能代理，能在复杂环境中自主完成任务，推动自动化革命。
Abstract
Recent web agents use world models for test-time action selection by sampling candidate actions, predicting the resulting web states, and ranking them with a ranker model or a Process Reward Model (PRM). These world models are typically trained via supervised next-state prediction to generate fixed representations like HTML or AXTree snapshots. However, this objective is misaligned with the downstream ranker, which relies on predicted states being discriminative across candidates to accurately score them. To address this, we introduce predicted-state matching, a training objective where the predicted representation must distinguish the true resulting state from those reached by alternative actions. We train these models using a branching web-agent dataset derived from WebArena Go-Browse trajectories, where every decision point contains multiple alternative actions and their resulting states. Experiments on our held-out predicted-state matching benchmark show that our approach outperforms world models trained with supervised next-state prediction. We further show that our approach improves PRM-style action ranking on WebPRMBench compared with action-only PRMs and PRMs augmented with supervised-next-state world models. Finally, on WebArena-Lite, using our world model for test-time action selection improves end-to-end task success. Our project page is available at: https://dhruvpendharkar.github.io/dwm/.