DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents
DeepResearch Bench offers 100 PhD-level tasks to evaluate deep research agents' report generation and information retrieval capabilities.
Key Findings
Methodology
The paper introduces DeepResearch Bench, a benchmark with 100 PhD-level tasks, using two novel methods: RACE and FACT. RACE evaluates report quality with dynamic weighting and adaptive criteria, while FACT assesses information retrieval by evaluating effective citation count and accuracy.
Key Results
- In the RACE framework, Gemini-2.5-Pro Deep Research achieved the highest report quality score of 48.88.
- In the FACT framework, Gemini-2.5-Pro Deep Research significantly led in effective citations, averaging 111.21.
- Perplexity Deep Research excelled in citation accuracy with a score of 90.24.
Significance
This research provides the first systematic benchmark for evaluating deep research agents, addressing the gap where existing frameworks fail to comprehensively assess agents' multifaceted capabilities. By open-sourcing the benchmark and evaluation protocols, it accelerates the development of practical LLM-based agents.
Technical Contribution
Introduced RACE and FACT frameworks for evaluating report generation quality and information retrieval capabilities. RACE offers a flexible evaluation method with dynamic weighting and adaptive criteria, while FACT evaluates citation accuracy and richness through automated steps.
Novelty
This is the first comprehensive benchmark for deep research agents, innovatively combining dynamic weighting and adaptive criteria to better reflect human judgment.
Limitations
- RACE framework may not fully capture subtle differences in report quality for certain tasks.
- FACT framework relies on Judge LLM's judgment, which may be limited by model performance.
Future Work
Future research could explore more complex tasks and multilingual support to further enhance the precision and applicability of agent evaluations.
AI Executive Summary
In today's information-rich era, deep research agents significantly enhance research efficiency by automating multistep web exploration and information retrieval. However, the lack of a systematic evaluation benchmark limits their further development. DeepResearch Bench addresses this issue by providing 100 PhD-level research tasks meticulously crafted by domain experts across 22 fields.
The paper introduces two novel methods to evaluate the capabilities of deep research agents. The RACE framework assesses report generation quality using dynamic weighting and adaptive criteria, while the FACT framework evaluates information retrieval capabilities by assessing effective citation count and accuracy. Experimental results show that Gemini-2.5-Pro Deep Research excels in both frameworks, particularly in effective citation count.
By open-sourcing DeepResearch Bench and key evaluation components, this research provides crucial support for the development of LLM-based agents. It not only fills the gap in existing frameworks but also offers new directions for future research, especially in more complex tasks and multilingual support.
Deep Analysis
Background
With the comprehensive advancement of large language models (LLMs), LLM-based agent systems are designed to tackle increasingly complex tasks. Deep Research Agents (DRAs) are among the most widely used agents, significantly enhancing productivity. However, comprehensively evaluating these DRAs presents substantial challenges, as their internal reasoning and information retrieval processes are not transparent, making the final generated report the primary interface for assessing overall performance.
Core Problem
Existing evaluation frameworks often fail to offer a dedicated assessment of agents' multifaceted capabilities, typically focusing on isolated capabilities such as web browsing and information retrieval, or generative abilities disconnected from real-time information acquisition. This evaluation gap limits the further development of deep research agents.
Innovation
DeepResearch Bench introduces RACE and FACT frameworks, providing the first comprehensive benchmark for deep research agents. RACE offers a flexible evaluation method with dynamic weighting and adaptive criteria, while FACT evaluates citation accuracy and richness through automated steps.
Methodology
- �� RACE Framework: Evaluates report quality using dynamic weighting and adaptive criteria. • FACT Framework: Assesses information retrieval capabilities by evaluating effective citation count and accuracy. • Dataset: Comprises 100 PhD-level research tasks across 22 fields. • Experiments: Evaluate four early-released deep research agents, comparing their performance in RACE and FACT frameworks.
Experiments
The experimental design includes using RACE and FACT frameworks to evaluate four early-released deep research agents: Gemini-2.5-Pro Deep Research, OpenAI Deep Research, Grok Deeper Search, and Perplexity Deep Research. Evaluation metrics include report quality, citation accuracy, and effective citation count.
Results
Experimental results show that Gemini-2.5-Pro Deep Research excels in both RACE and FACT frameworks, particularly in effective citation count, averaging 111.21. Perplexity Deep Research excels in citation accuracy with a score of 90.24.
Applications
The benchmark can be used to evaluate and improve deep research agents' applications across multiple fields, such as academic research, market analysis, and technical report generation. By enhancing the precision and applicability of agent evaluations, it better meets the needs of real-world applications.
Limitations & Outlook
The RACE framework may not fully capture subtle differences in report quality for certain tasks, while the FACT framework relies on Judge LLM's judgment, which may be limited by model performance. Future research could explore more complex tasks and multilingual support.
Plain Language Accessible to non-experts
Imagine you're in a massive library searching for information. Traditionally, you'd have to go to each shelf, find books, read, and summarize information yourself, which could take hours. A deep research agent is like a super librarian that can automatically find relevant books, extract key information, and generate a detailed report for you in minutes. DeepResearch Bench is designed to evaluate the efficiency and accuracy of these super librarians. Through a series of carefully designed tasks and evaluation criteria, we can assess how well these agents perform across different fields, helping them serve us better.
ELI14 Explained like you're 14
Imagine you're playing a big online game and need to quickly find hidden treasures. The traditional way is to search the map yourself, which could take a long time. A deep research agent is like a game super helper that can find all the hidden treasures in minutes and tell you how to get them. DeepResearch Bench evaluates these super helpers to see how fast and accurately they can find treasures. With this benchmark, we can know which helper performs best and help them become even stronger!
Glossary
Deep Research Agent
An agent based on large language models that automates multistep web exploration and information retrieval.
Used to generate analyst-grade reports.
RACE Framework
A method for evaluating report generation quality using dynamic weighting and adaptive criteria.
Used to assess research reports generated by agents.
FACT Framework
A method for evaluating information retrieval capabilities by assessing effective citation count and accuracy.
Used to evaluate agents' information retrieval and citation capabilities.
Effective Citation
A citation in a report that is verified to accurately support a statement.
Used to evaluate agents' information retrieval capabilities.
Dynamic Weighting
Task-specific weights generated for evaluating report quality.
Used in the RACE framework for assessing report generation quality.
Open Questions Unanswered questions from this research
- 1 How to improve evaluation precision of deep research agents in multilingual environments?
- 2 What are the limitations of current frameworks in handling more complex tasks?
Applications
Immediate Applications
Academic Research
Researchers can use deep research agents to quickly generate literature reviews and research reports, enhancing research efficiency.
Long-term Vision
Automated Market Analysis
Businesses can use deep research agents for market trend analysis and competitor research, optimizing business decisions.
Abstract
Deep Research Agents are a prominent category of LLM-based agents. By autonomously orchestrating multistep web exploration, targeted retrieval, and higher-order synthesis, they transform vast amounts of online information into analyst-grade, citation-rich reports--compressing hours of manual desk research into minutes. However, a comprehensive benchmark for systematically evaluating the capabilities of these agents remains absent. To bridge this gap, we present DeepResearch Bench, a benchmark consisting of 100 PhD-level research tasks, each meticulously crafted by domain experts across 22 distinct fields. Evaluating DRAs is inherently complex and labor-intensive. We therefore propose two novel methodologies that achieve strong alignment with human judgment. The first is a reference-based method with adaptive criteria to assess the quality of generated research reports. The other framework is introduced to evaluate DRA's information retrieval and collection capabilities by assessing its effective citation count and overall citation accuracy. We have open-sourced DeepResearch Bench and key components of these frameworks at https://github.com/Ayanami0730/deep_research_bench to accelerate the development of practical LLM-based agents.