CorpusQA: A 10 Million Token Benchmark for Corpus-Level Analysis and Reasoning
CorpusQA benchmark challenges LLMs' global reasoning over 10 million tokens.
Key Findings
Methodology
CorpusQA generates complex queries via a data synthesis framework, ensuring programmatic ground-truth answers, challenging systems to perform holistic reasoning over vast unstructured text. It uses a multi-stage pipeline including document collection, schema extraction, query generation, and NL2SQL execution.
Key Results
- At 128K tokens, the Gemini-2.5-Pro model performed best with 80.19% accuracy, but significantly dropped to 50.73% at 1M tokens.
- Memory-augmented systems maintained 11.13% accuracy at 10M tokens, while RAG systems nearly failed.
- Experiments show significant limitations of current LLMs in ultra-long text processing; memory-augmented architectures show more promise.
Significance
This research fills the gap in existing benchmarks for corpus-level analysis, advancing the evaluation of LLMs' long-context reasoning capabilities. It reveals the shortcomings of current systems in handling dispersed evidence, emphasizing the need for developing new architectures.
Technical Contribution
Introduced a novel data synthesis framework capable of generating complex queries with verifiable answers, overcoming limitations of existing benchmarks. Demonstrated that memory-augmented architectures are more robust for large-scale corpus processing.
Novelty
First to propose a benchmark for corpus-level analysis at 10 million tokens, challenging existing LLMs' capabilities in global information integration, contrasting sharply with traditional RAG methods.
Limitations
- Current LLMs show significant performance degradation when processing ultra-long texts, especially beyond 1M tokens.
- RAG systems nearly fail when evidence is highly dispersed.
Future Work
Future research could explore more efficient memory-augmented architectures, developing new methods for effective information integration in ultra-long texts.
AI Executive Summary
As large language models (LLMs) advance in handling long texts, existing benchmarks fall short in evaluating their reasoning capabilities across entire document repositories. CorpusQA addresses this gap by introducing a new benchmark scaling up to 10 million tokens. This benchmark generates complex queries through a novel data synthesis framework, requiring systems to perform holistic reasoning over vast unstructured text without relying on fallible human annotation.
Experimental results reveal that even state-of-the-art long-context LLMs struggle as input length increases, and standard retrieval-augmented generation systems collapse entirely. Memory-augmented agentic architectures offer a more robust alternative, suggesting a critical shift is needed from simply extending context windows to developing advanced architectures for global information synthesis.
The significance of this research lies in advancing the evaluation of LLMs' long-context reasoning capabilities, revealing the shortcomings of current systems in handling dispersed evidence, and emphasizing the need for developing new architectures. Future research could explore more efficient memory-augmented architectures, developing new methods for effective information integration in ultra-long texts.
Deep Analysis
Background
In recent years, as large language models (LLMs) have advanced in handling long texts, existing benchmarks have fallen short in evaluating their reasoning capabilities across entire document repositories. Most existing benchmarks are limited to single long texts or rely on a 'sparse retrieval' assumption, which fails in corpus-level analysis where evidence is highly dispersed.
Core Problem
Existing benchmarks cannot effectively evaluate LLMs' reasoning capabilities when processing large-scale corpora, especially when evidence is dispersed and requires global integration. This is crucial in high-stakes professional domains such as finance, law, and scientific research.
Innovation
CorpusQA generates complex queries via a novel data synthesis framework, requiring systems to perform holistic reasoning over vast unstructured text without relying on fallible human annotation. This framework decouples reasoning from textual representation, creating complex, computation-intensive queries.
Methodology
- �� Document collection and filtering to ensure long and data-rich inputs
- �� Schema extraction using a multi-model voting approach to ensure data quality
- �� Query generation via templates and LLM-based expansion to create diverse natural language queries
- �� Data table aggregation to construct a global data table as a knowledge base
- �� NL2SQL execution to ensure verifiable answers for each query
- �� Final QA pair assembly to challenge models to perform reasoning over raw text
Experiments
The experimental design includes evaluating a wide array of state-of-the-art long-context LLMs, RAG systems, and more advanced agentic architectures. Results show that even state-of-the-art LLMs struggle as input length increases, and standard RAG systems collapse entirely.
Results
At 128K tokens, the Gemini-2.5-Pro model performed best with 80.19% accuracy, but significantly dropped to 50.73% at 1M tokens. Memory-augmented systems maintained 11.13% accuracy at 10M tokens, while RAG systems nearly failed.
Applications
Applications include financial analysis, legal research, and scientific research, enabling analysts to perform global reasoning and information integration across large document repositories.
Limitations & Outlook
Current LLMs show significant performance degradation when processing ultra-long texts, especially beyond 1M tokens. RAG systems nearly fail when evidence is highly dispersed. Future research could explore more efficient memory-augmented architectures, developing new methods for effective information integration in ultra-long texts.
Plain Language Accessible to non-experts
Imagine you're in a massive library searching for a specific book, but none of the books have an index. CorpusQA is like a super-smart librarian who not only helps you find the book but also extracts relevant information from the entire library and integrates it into a complete answer. It doesn't rely on just a few books but gathers information from the entire library. It's like finding all the right puzzle pieces from a huge pile and putting them together to form a complete picture.
ELI14 Explained like you're 14
Imagine you're playing a super complex puzzle game. You have a huge pile of puzzle pieces but no picture on the box to guide you. CorpusQA is like a super-smart helper that finds all the right pieces from the pile and puts them together to form a complete picture. It doesn't just look at a few pieces but finds the right ones from all of them. It's like finding the right answer in a giant puzzle game!
Glossary
CorpusQA
A benchmark for evaluating LLMs' reasoning capabilities over large-scale corpora.
Used to test LLMs' performance on processing 10 million tokens of unstructured text.
RAG (Retrieval-Augmented Generation)
A method that generates answers by retrieving relevant document chunks.
Performs poorly when evidence is highly dispersed.
LLM (Large Language Model)
Models capable of handling long texts and performing complex reasoning.
Used to test reasoning capabilities over large-scale corpora.
Memory-Augmented Architecture
An architecture that enhances a model's ability to process ultra-long texts through memory mechanisms.
Performs well in large-scale corpus processing.
Data Synthesis Framework
A method for generating complex queries with verifiable answers.
Used to create the benchmark data for CorpusQA.
Open Questions Unanswered questions from this research
- 1 How to effectively integrate dispersed information in ultra-long texts? Current methods perform poorly when evidence is dispersed, requiring new architectures.
- 2 How to improve LLMs' efficiency in processing ultra-long texts? Current models show significant performance degradation beyond 1M tokens.
Applications
Immediate Applications
Financial Analysis
Helps analysts perform global reasoning and information integration across large financial reports, improving decision-making efficiency.
Legal Research
Supports legal researchers in extracting and integrating relevant information from vast case law databases, enhancing legal analysis capabilities.
Long-term Vision
Scientific Research
Enables global analysis of large-scale literature in scientific research, advancing interdisciplinary research progress.
Abstract
While large language models now handle million-token contexts, their capacity for reasoning across entire document repositories remains largely untested. Existing benchmarks are inadequate, as they are mostly limited to single long texts or rely on a "sparse retrieval" assumption-that answers can be derived from a few relevant chunks. This assumption fails for true corpus-level analysis, where evidence is highly dispersed across hundreds of documents and answers require global integration, comparison, and statistical aggregation. To address this critical gap, we introduce CorpusQA, a new benchmark scaling up to 10 million tokens, generated via a novel data synthesis framework. By decoupling reasoning from textual representation, this framework creates complex, computation-intensive queries with programmatically guaranteed ground-truth answers, challenging systems to perform holistic reasoning over vast, unstructured text without relying on fallible human annotation. We further demonstrate the utility of our framework beyond evaluation, showing that fine-tuning on our synthesized data effectively enhances an LLM's general long-context reasoning capabilities. Extensive experiments reveal that even state-of-the-art long-context LLMs struggle as input length increases, and standard retrieval-augmented generation systems collapse entirely. Our findings indicate that memory-augmented agentic architectures offer a more robust alternative, suggesting a critical shift is needed from simply extending context windows to developing advanced architectures for global information synthesis.