Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
Introduces Mind2Web 2 benchmark with Agent-as-a-Judge, evaluating 130 long-horizon tasks achieving 50-70% of human performance.
Key Findings
Methodology
This paper proposes a tree-structured rubric-based Agent-as-a-Judge framework, constructing task-specific judge agents that automatically assess answer correctness and source attribution. Using large models like GPT-4, it performs information extraction and verification across hierarchical nodes, enabling reliable evaluation of complex, time-varying answers. The benchmark includes 130 high-quality, real-world tasks across domains, with over 1,000 hours of expert labor. Evaluation of ten frontier systems and human performance shows OpenAI Deep Research achieves 50-70% of human accuracy with half the time, demonstrating significant potential.
Key Results
- OpenAI Deep Research attains 50-70% accuracy in complex long-horizon tasks, with an average time half of human effort, outperforming previous systems.
- The framework effectively handles time-varying information and source attribution, validated by comprehensive experiments.
- Automated, hierarchical evaluation proves reliable, establishing a new paradigm for long-term, dynamic answer assessment.
Significance
This work addresses the critical gap in evaluating long-horizon, dynamic information retrieval systems, providing a rigorous, automated assessment tool. It enhances trustworthiness and practical deployment of AI in complex real-world scenarios, impacting academia and industry by enabling more reliable system development.
Technical Contribution
The paper introduces a novel tree-structured rubric and an automated judge agent leveraging large models for multi-layered information extraction and verification. This approach significantly improves evaluation efficiency and accuracy for complex, evolving tasks, surpassing static, single-answer methods. It also automates rubric generation and scoring, reducing manual effort and enabling scalable assessments.
Novelty
First to systematically implement an Agent-as-a-Judge framework with hierarchical rubrics for long-horizon, real-time web tasks. It combines automated rubric creation, multi-source verification, and multi-step evaluation, filling a major gap in the field of dynamic answer assessment.
Limitations
- The pre-defined rubric tree may not generalize well to extremely complex or ambiguous tasks, requiring manual adjustments.
- Expert labor is still needed for task validation and script development, limiting scalability.
- Model errors in extreme time variation or conflicting sources can lead to misjudgments, necessitating robustness improvements.
Future Work
Future directions include developing adaptive, self-generating rubrics via reinforcement learning, expanding multi-modal verification, and scaling to larger datasets. Enhancing robustness against ambiguous or conflicting information remains a priority, aiming for fully autonomous, end-to-end evaluation pipelines.
AI Executive Summary
The rapid evolution of AI-driven web search systems has introduced a new paradigm—agentic search—that autonomously browses, synthesizes, and answers complex queries. Unlike traditional search engines, these systems perform multi-step, long-horizon tasks involving real-time web interactions, posing significant challenges for evaluation. Existing benchmarks focus on short, static answers, insufficient for assessing the capabilities of advanced agentic systems in dynamic environments.
To address this gap, the authors present Mind2Web 2, a comprehensive benchmark comprising 130 real-world, high-quality long-horizon tasks across diverse domains. Each task involves extensive web browsing and information synthesis, validated through expert labor exceeding 1,000 hours. The benchmark emphasizes realism, complexity, and verifiability, reflecting practical user needs.
Central to this work is the innovative Agent-as-a-Judge framework, which employs a hierarchical, tree-structured rubric to automatically evaluate answers. This approach leverages large language models like GPT-4 for information extraction and source verification, enabling scalable, reliable assessment of answers that evolve over time. The framework's effectiveness is demonstrated through systematic evaluation of ten state-of-the-art systems and human performance, with the best system, OpenAI Deep Research, achieving 50-70% of human accuracy while reducing time expenditure by half.
This research marks a significant step forward in AI evaluation methodology, facilitating the development of more trustworthy, capable long-horizon web agents. It opens avenues for future work on adaptive rubrics, multi-modal verification, and broader application scenarios. Despite current limitations, such as dependence on predefined rubrics and expert labor, the framework sets a new standard for dynamic, real-world AI system assessment, promising substantial impact across academia and industry.
Deep Analysis
Background
Web search技术经历了从TF-IDF、PageRank到学习排序的逐步演进，极大提升了信息检索效率。然而，用户仍需手动筛选和整合网页信息，面对复杂问题时负担沉重。大规模语言模型（如GPT系列）推动了代理式搜索的发展，系统能自主浏览网页、整合信息，显著提升效率。现有评估多集中于短时、静态任务，缺乏对长时、动态场景的系统性衡量，限制了技术的推广应用。
Core Problem
当前评估体系难以应对长时、动态、多源信息交互的复杂任务。传统指标多依赖单一答案或静态网页，无法反映系统在真实环境中的表现。长时任务涉及多步骤、多网页、多信息源，答案变化频繁，评估难度大，亟需新的自动化、可靠的评估方法以推动技术发展。
Innovation
本文提出基于树状结构的Agent-as-a-Judge框架，自动生成任务特定的评分标准，结合大模型实现信息提取与验证，支持长时、动态任务的自动评估。创新点包括：1）多层次评分树设计，确保评判的细粒度和全面性；2）自动化脚本生成与自我优化，降低人工成本；3）多源信息验证机制，提升答案的可信度。这些创新突破了静态、单一答案评估的局限，适应复杂场景。
Methodology
- �� 构建130个真实长时任务，涵盖多领域，专家劳动超过千小时。• 设计树状评分标准，定义每个节点的评判条件，确保评估的细粒度与层次性。• 利用大模型（如GPT-4）实现信息提取（Extractor）和验证（Verifier），自动执行评分。• 生成评判脚本，通过自我调试和专家验证，确保脚本的准确性和通用性。• 评估十个系统与人类表现，统计准确率、耗时等指标，分析系统优劣。• 采用Pass@3等指标，衡量多次尝试成功率，确保评估的稳健性。
Experiments
采用真实长时任务集，比较多种系统（如OpenAI Deep Research、Perplexity Search等）与人类表现。指标包括Partial Completion、Success Rate、耗时和答案长度。通过多轮评估，验证系统在时间变化、信息验证方面的能力。设置不同参数（如搜索深度、模型版本），进行消融分析，确保结果的可靠性。实验还包括对评判脚本的准确性验证，确保自动评估的可信度。
Results
OpenAI Deep Research在长时任务中达成50-70%的准确率，平均耗时为人类一半，显著优于其他系统。系统在时间变化和信息来源归属方面表现优异，验证了Agent-as-a-Judge的有效性。评估结果表明，长时、多源、多步骤任务中，系统仍有提升空间，但已有明显突破，为未来发展提供方向。
Applications
该评估框架可用于开发更智能、更可信的长时域搜索系统，广泛应用于科研、商务、法律等领域的复杂信息检索。未来，结合多模态数据和强化学习，将推动系统在实时决策、知识更新等方面的能力提升，带来行业变革。
Limitations & Outlook
目前评判脚本依赖预定义评分树，难以应对极端复杂或模糊任务。专家劳动成本高，扩展性有限。模型在极端时间变化或多源信息冲突时可能误判，需优化鲁棒性。未来需开发自适应、可扩展的评估机制，以应对更复杂场景。
Plain Language Accessible to non-experts
想象你在一家大型工厂工作，工厂里有许多不同的机器和流程，每天都在不断变化。你需要确保每个产品都符合标准，流程顺畅。传统方法就像用一只手工检查每个产品，费时又容易出错。而现在，有了智能机器人，它可以自动检测每个环节，快速判断产品是否合格，还能追溯每个部分的来源。这个机器人就像论文中的Agent-as-a-Judge，它用一套规则（评分树）来判断答案的正确性和来源，确保每个环节都符合要求。这样，工厂的效率大大提高，产品质量也更有保障。这个比喻帮助我们理解，复杂的任务需要智能系统像工厂里的机器人一样，自动、准确地完成评估工作。
ELI14 Explained like you're 14
想象你在学校里参加一个超级难的寻宝游戏，你要找到很多线索，最后拼出答案。这个游戏很复杂，有很多步骤，要去不同的地方，找到不同的线索，还要确认每个线索是不是正确的。以前，我们只能自己一个一个去找，花很多时间，也容易搞错。现在，有个聪明的机器人助手，它可以帮你检查每个线索是不是正确，还能告诉你每个线索来自哪里。这个助手用了一套特别的规则，就像游戏里的评分标准，确保每个线索都是真的、可靠的。这样，你就能更快、更准确地完成任务，而且还不用担心出错了。这就像论文中的方法，用智能系统帮忙评估复杂的长时间任务，让工作变得更简单、更可靠。
Glossary
Agent-as-a-Judge (代理评判)
一种利用大模型自动评估复杂答案正确性和来源归属的方法，基于树状评分标准实现多层次判断。
论文提出的自动评估框架核心技术。
树状评分标准 (Tree-structured Rubric)
一种将评判标准层级化、结构化的评分体系，每个节点代表一个评判条件，最终汇总得出整体评价。
用于自动化、多源、多步骤答案评估。
长时任务 (Long-horizon Tasks)
需要多步骤、多网页、多信息源交互，耗时较长的复杂任务，超出传统短时评估范围。
本文评估的主要对象。
信息提取 (Information Extraction)
利用大模型自动从答案文本中提取结构化信息，如网址、价格等。
评判脚本中的关键步骤。
验证机制 (Verification Mechanism)
通过模型和网页截图验证信息的真实性和来源归属，确保答案可靠。
保证答案正确性的重要环节。
Open Questions Unanswered questions from this research
- 1 如何进一步提升评判代理在极端复杂或模糊任务中的鲁棒性？
- 2 多源、多模态信息的验证机制如何结合深度学习进行优化？
- 3 未来能否实现端到端自动化的长时任务评估体系？
Applications
Immediate Applications
智能信息检索系统
可用于科研、法律、商务等领域的长时信息搜集与验证，提升效率与可信度。
自动化内容审核
在内容生成平台自动评估内容的真实性与来源，确保信息质量。
Long-term Vision
智能决策支持
结合长时任务评估，推动自动化决策系统在金融、医疗等行业的应用，实现自主决策。
Abstract
Agentic search such as Deep Research systems-where agents autonomously browse the web, synthesize information, and return comprehensive citation-backed answers-represents a major shift in how users interact with web-scale information. While promising greater efficiency and cognitive offloading, the growing complexity and open-endedness of agentic search have outpaced existing evaluation benchmarks and methodologies, which largely assume short search horizons and static answers. In this paper, we introduce Mind2Web 2, a benchmark of 130 realistic, high-quality, and long-horizon tasks that require real-time web browsing and extensive information synthesis, constructed with over 1000 hours of human labor. To address the challenge of evaluating time-varying and complex answers, we propose a novel Agent-as-a-Judge framework. Our method constructs task-specific judge agents based on a tree-structured rubric design to automatically assess both answer correctness and source attribution. We conduct a comprehensive evaluation of ten frontier agentic search systems and human performance, along with a detailed error analysis to draw insights for future development. The best-performing system, OpenAI Deep Research, can already achieve 50-70% of human performance while spending half the time, highlighting its great potential. Altogether, Mind2Web 2 provides a rigorous foundation for developing and benchmarking the next generation of agentic search systems.