NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities
NeedleBench evaluates LLMs' retrieval and reasoning across sparse and dense contexts, revealing weaknesses in dense reasoning tasks.
Key Findings
Methodology
NeedleBench creates synthetic tasks: sparse tasks (e.g., Single-Needle Retrieval, Multi-Needle Retrieval) and dense tasks (Ancestral Trace Challenge). Sparse tasks test retrieval with minimal relevant content, while dense tasks evaluate reasoning with distributed critical information.
Key Results
- Result 1: At 32K context length, Qwen-2.5-72B achieved 100% accuracy in Single-Needle Retrieval, while older models like Zephyr-7B-Beta scored only 36.06%.
- Result 2: In dense tasks, leading models like DeepSeek-R1 underperformed, with reasoning accuracy below 50%.
- Result 3: Identified 'under-thinking,' where models prematurely conclude reasoning despite incomplete information processing.
Significance
NeedleBench avoids interference from pretraining knowledge by using synthetic data, providing fairer evaluation. Its dense tasks address gaps in assessing complex reasoning, critical for fields like law and medicine.
Technical Contribution
Introduced a flexible framework supporting context lengths from 4K to 1M+. Designed the Ancestral Trace Challenge to simulate multi-step reasoning, with fine-grained metrics like Effective Needle Length (ENL-50).
Novelty
First systematic evaluation of LLMs on dense reasoning tasks, introducing 'under-thinking' and leveraging synthetic data to eliminate pretraining bias.
Limitations
- Limitation 1: Synthetic data may not fully represent real-world scenarios, affecting generalizability.
- Limitation 2: Focused on text-based reasoning, lacking multimodal evaluations.
- Limitation 3: Results constrained by models' context window sizes, limiting full potential assessment.
Future Work
Future work could expand to multimodal tasks, improve dense reasoning capabilities, and optimize task generation for real-world alignment.
AI Executive Summary
Long-context processing is vital for LLM applications in fields like law and healthcare. Existing evaluations either rely on real-world texts, risking interference from pretraining knowledge, or use irrelevant filler to extend context, reducing effectiveness.
NeedleBench introduces synthetic tasks: sparse tasks (e.g., Single-Needle Retrieval) and dense tasks (Ancestral Trace Challenge). Experiments show that while models like Qwen-2.5 excel in retrieval, they struggle with dense reasoning, often exhibiting 'under-thinking,' where reasoning is prematurely concluded.
NeedleBench provides a critical tool for evaluating LLMs' long-context capabilities, addressing gaps in dense task assessment. Future research can extend to multimodal tasks and optimize task generation for real-world applications.
Deep Analysis
Background
Long-context capabilities are essential for LLMs in domains like legal analysis and medical records. Early methods like Needle In A Haystack tested retrieval but focused on sparse tasks, failing to assess complex reasoning.
Core Problem
Existing benchmarks lack methods to evaluate dense reasoning tasks and often rely on real-world texts, making them susceptible to pretraining knowledge interference.
Innovation
NeedleBench introduces synthetic sparse and dense tasks, with the Ancestral Trace Challenge providing the first systematic evaluation of dense reasoning performance.
Methodology
- �� Sparse tasks: Insert minimal key information to test retrieval.
- �� Dense tasks: Ancestral Trace Challenge distributes critical information to test reasoning.
- �� Metrics: Introduced ENL-50 to quantify reasoning depth in complex tasks.
Experiments
Evaluated mainstream models like Qwen-2.5 and DeepSeek-R1 at 32K and 128K context lengths, assessing retrieval and reasoning performance.
Results
Qwen-2.5 achieved 100% accuracy in Single-Needle Retrieval but underperformed in dense tasks, with ENL-50 below 512 needles.
Applications
Applicable for evaluating long-context reasoning in legal and medical domains, especially for complex document analysis.
Limitations & Outlook
Synthetic data may lack real-world fidelity, and multimodal tasks remain unexplored. Future work should optimize task generation.
Plain Language Accessible to non-experts
Imagine finding a book in a library (sparse task) versus solving a complex family tree puzzle within the book (dense task). Sparse tasks test simple retrieval, while dense tasks require integrating all clues.
ELI14 Explained like you're 14
Think of a treasure hunt: sparse tasks are like finding a map hidden in a room, while dense tasks are solving the map's riddles to locate the treasure! Fun, right?
Glossary
NeedleBench
A framework for evaluating LLMs' long-context capabilities, including sparse and dense tasks.
Used to test retrieval and reasoning.
Dense Task
Tasks where every sentence contains critical information for the question.
Ancestral Trace Challenge is a dense task.
ENL-50
Effective Needle Length, measuring the maximum task complexity a model handles with ≥50% accuracy.
Quantifies reasoning depth.
Under-thinking
Phenomenon where models prematurely conclude reasoning despite incomplete information.
Observed in dense tasks.
Synthetic Data
Artificially generated data to avoid pretraining knowledge interference.
Used in all NeedleBench tasks.
Open Questions Unanswered questions from this research
- 1 How to design synthetic data closer to real-world scenarios?
- 2 How to extend NeedleBench to multimodal tasks?
Applications
Immediate Applications
Legal Document Analysis
Evaluate models' ability to extract key clauses from lengthy legal texts.
Medical Record Processing
Test models' integration of multi-segment patient records.
Long-term Vision
Multimodal Reasoning
Extend to tasks combining text and images, enabling cross-domain applications.
Abstract
The capability of large language models to handle long-context information is crucial across various real-world applications. Existing evaluation methods often rely either on real-world long texts, making it difficult to exclude the influence of models' inherent knowledge, or introduce irrelevant filler content to artificially achieve target lengths, reducing assessment effectiveness. To address these limitations, we introduce NeedleBench, a synthetic framework for assessing retrieval and reasoning performance in bilingual long-context tasks with adaptive context lengths. NeedleBench systematically embeds key data points at varying depths to rigorously test model capabilities. Tasks are categorized into two scenarios: information-sparse, featuring minimal relevant details within extensive irrelevant text to simulate simple retrieval tasks; and information-dense (the Ancestral Trace Challenge), where relevant information is continuously distributed throughout the context to simulate complex reasoning tasks. Our experiments reveal that although recent reasoning models like Deepseek-R1 and OpenAI's o3 excel in mathematical reasoning, they struggle with continuous retrieval and reasoning in information-dense scenarios, even at shorter context lengths. We also characterize a phenomenon termed 'under-thinking', where models prematurely conclude reasoning despite available information. NeedleBench thus provides critical insights and targeted tools essential for evaluating and improving LLMs' long-context capabilities. All resources are available at OpenCompass: https://github.com/open-compass/opencompass.