Benchmarking Deep Search over Heterogeneous Enterprise Data
HERB benchmark reveals retrieval bottlenecks in RAG systems over heterogeneous enterprise data, scoring 32.96 on average.
Key Findings
Methodology
The study employs the HERB benchmark, simulating enterprise workflows to generate synthetic data for evaluating RAG systems. This involves multi-source, multi-hop reasoning across documents, meeting transcripts, Slack messages, and other heterogeneous data sources. A synthetic data pipeline generates 39,190 enterprise artifacts for fine-grained evaluation.
Key Results
- Even the best-performing RAG methods achieve an average score of only 32.96 on the HERB benchmark, indicating difficulties in conducting deep searches and retrieving all necessary evidence.
- The ReAct agent shows significant improvement by combining retrieval and reasoning, increasing the score by 12.35 points to 32.96.
- Gemini-2.5-Flash performs best in long-context reasoning, scoring 76.55, but still has room for improvement.
Significance
This research provides a new benchmark for RAG systems, addressing the gap in deep search over heterogeneous enterprise data. By simulating real enterprise environments, the HERB benchmark reveals the limitations of current RAG methods in handling complex enterprise queries, particularly in retrieval and multi-hop reasoning.
Technical Contribution
The HERB benchmark offers a unique evaluation platform by generating synthetic enterprise data and multi-hop questions. It emphasizes the retrieval and reasoning capabilities of RAG systems over heterogeneous data sources, advancing research in long-context reasoning.
Novelty
HERB is the first RAG benchmark to support implicit reasoning across heterogeneous data formats. Unlike existing datasets, it integrates both structured and unstructured data, enabling deeper reasoning.
Limitations
- Existing RAG methods struggle to retrieve all necessary evidence, leading to reasoning over partial context and significant performance degradation.
- Systems show inconsistent performance in identifying unanswerable queries, indicating room for improvement.
Future Work
Future research could explore improving retrieval systems to support deep search and developing more effective multi-hop reasoning methods. Further optimizing long-context reasoning capabilities is also a key direction.
AI Executive Summary
In modern enterprises, the ability to handle heterogeneous data sources is crucial. However, existing RAG systems perform poorly when dealing with complex enterprise queries. To address this, researchers developed the HERB benchmark, simulating enterprise workflows to generate synthetic data for evaluating RAG system performance. The HERB benchmark reveals the limitations of current methods in deep search and retrieving all necessary evidence, particularly in multi-hop reasoning.
The study shows that even the most advanced RAG methods achieve an average score of only 32.96 on the HERB benchmark, far below expectations. The ReAct agent significantly improves performance by combining retrieval and reasoning, demonstrating potential in complex enterprise queries. The evaluation of long-context reasoning also highlights significant challenges, with Gemini-2.5-Flash performing best but still having room for improvement.
The introduction of the HERB benchmark provides a new perspective for evaluating RAG systems, emphasizing the need for deep search over heterogeneous data sources. Future research directions include improving retrieval systems and multi-hop reasoning methods to enhance RAG system performance in complex enterprise environments.
Deep Analysis
Background
In recent years, RAG systems have gained widespread adoption in enterprise applications, particularly for tasks requiring grounded responses. However, existing multi-hop RAG benchmarks are typically limited to document clusters, lacking deep search capabilities over heterogeneous data sources. The HERB benchmark addresses this gap by simulating enterprise workflows and generating synthetic data.
Core Problem
Existing RAG methods face challenges in handling heterogeneous enterprise data, particularly in retrieving all necessary evidence for multi-hop reasoning. The retrieval bottleneck leads to reasoning over partial context, significantly reducing performance.
Innovation
The HERB benchmark provides a unique evaluation platform by generating synthetic enterprise data and multi-hop questions. It integrates both structured and unstructured data, enabling deeper reasoning and filling the gap in existing RAG benchmarks.
Methodology
- �� Simulate enterprise workflows using a synthetic data pipeline
- �� Generate 39,190 enterprise artifacts, covering documents, meeting transcripts, Slack messages, etc.
- �� Design multi-hop questions for fine-grained evaluation
- �� Evaluate RAG systems' retrieval and reasoning capabilities
Experiments
Experiments use the HERB benchmark to evaluate various RAG systems and LLM configurations. The dataset includes 39,190 enterprise artifacts and 815 answerable queries. Evaluation metrics include average performance score and the ability to identify unanswerable queries.
Results
Results show that even the best-performing RAG methods achieve an average score of only 32.96. The ReAct agent significantly improves performance by combining retrieval and reasoning. The evaluation of long-context reasoning also highlights significant challenges.
Applications
The HERB benchmark can be used to evaluate RAG system performance in complex enterprise environments, helping to develop more effective retrieval and reasoning methods and improve enterprise data processing capabilities.
Limitations & Outlook
Existing RAG methods struggle to retrieve all necessary evidence, leading to reasoning over partial context. Additionally, systems show inconsistent performance in identifying unanswerable queries, indicating room for improvement.
Plain Language Accessible to non-experts
Imagine you're working in a large company, dealing with a lot of documents, meeting notes, and chat logs every day. The HERB benchmark is like a smart assistant that helps you find the answers you need from all this information. It simulates enterprise workflows to generate synthetic data and evaluate RAG system performance. Although current methods perform poorly in handling this complex data, the HERB benchmark provides new perspectives for improving these systems.
ELI14 Explained like you're 14
Imagine you're doing a big project at school, needing to find answers from different books, notes, and online resources. The HERB benchmark is like a super search engine that helps you find all the relevant information. While current methods perform poorly in handling this complex data, the HERB benchmark provides new perspectives for improving these systems. In the future, we can expect smarter systems to help us handle this complex information!
Glossary
RAG (Retrieval-Augmented Generation)
A technique combining retrieval and generation to find and generate answers from large data sets.
Used in the paper to evaluate system performance on enterprise data.
HERB (Heterogeneous Enterprise RAG Benchmark)
A new benchmark for evaluating RAG systems on heterogeneous enterprise data.
Used to simulate enterprise workflows and generate synthetic data.
Multi-hop Reasoning
A reasoning method requiring inference across multiple information sources to draw conclusions.
Crucial for handling complex enterprise queries.
Synthetic Data Pipeline
A process for generating simulated data to create evaluation benchmarks.
Used to generate enterprise data in the HERB benchmark.
ReAct Agent
An agent method combining retrieval and reasoning, enhancing RAG system performance.
Performed well in experiments, significantly improving performance.
Open Questions Unanswered questions from this research
- 1 How to improve RAG systems' retrieval capabilities to support deeper searches? Current methods perform poorly in handling complex enterprise queries.
- 2 How to enhance systems' ability to identify unanswerable queries? Current methods show inconsistent performance in this area.
Applications
Immediate Applications
Enterprise Data Processing
The HERB benchmark can be used to evaluate and improve enterprise data processing systems, helping companies handle complex queries more effectively.
Long-term Vision
Intelligent Information Retrieval
In the future, improved RAG systems could enable more intelligent information retrieval across a wider range of fields, enhancing efficiency across industries.
Abstract
We present a new benchmark for evaluating Deep Search--a realistic and complex form of retrieval-augmented generation (RAG) that requires source-aware, multi-hop reasoning over diverse, sparsed, but related sources. These include documents, meeting transcripts, Slack messages, GitHub, and URLs, which vary in structure and often contain human-to-human interactions. We build it using a synthetic data pipeline that simulates business workflows across product planning, development, and support stages, generating interconnected content with realistic noise and multi-hop questions with guaranteed ground-truth answers. We release our benchmark with both answerable and unanswerable queries, and retrieval pool of 39,190 enterprise artifacts, enabling fine-grained evaluation of long-context LLM and RAG systems. Our experiments reveal that even the best-performing agentic RAG methods achieve an average performance score of 32.96 on our benchmark. With further analysis, we highlight retrieval as the main bottleneck: existing methods struggle to conduct deep searches and retrieve all necessary evidence. Consequently, they often reason over partial context, leading to significant performance degradation.