Evidence-State Rewards for Long-Context Reasoning
MAVEN framework enhances long-context reasoning by rewarding dynamic evidence state transitions, improving performance by 3.5% on LongBench v2.
Key Findings
Methodology
MAVEN is a reinforcement learning framework with an editable evidence memory. It defines an answer-conditioned evidence-state value and rewards action-level state transitions: add actions are credited by marginal gain and hindsight contribution, link actions by evidence synergy, and drop actions by improved answer support after removing misleading evidence.
Key Results
- On LongBench v2, MAVEN improved the Llama-3.1-8B model by 3.5 percentage points, outperforming outcome-only RL and evidence identification baselines.
- MAVEN also excelled on LongReason and RULER datasets, significantly increasing evidence sufficiency and reducing distractor retention.
- At 128K context length, MAVEN performed exceptionally on the Qwen3-30B-A3B model, with an average score increase of 4.0 percentage points.
Significance
MAVEN significantly enhances long-context reasoning performance by optimizing stateful evidence navigation rather than one-shot evidence extraction. This method is not only academically significant but also provides more efficient information processing and reasoning tools for the industry, addressing long-standing challenges in effective reasoning over long texts.
Technical Contribution
MAVEN introduces an editable evidence memory and action-level reward mechanism, offering more granular process supervision compared to existing methods. Its uniqueness lies in incorporating dynamic changes in evidence state into the reward system, providing new theoretical guarantees and engineering possibilities.
Novelty
MAVEN is the first framework to incorporate dynamic changes in evidence state into the reward mechanism, providing more comprehensive process supervision and more efficient reasoning capabilities compared to traditional evidence extraction methods.
Limitations
- MAVEN may face high computational costs when handling extremely long contexts, especially when frequent updates to the evidence memory are required.
- In some complex reasoning tasks, more manual tuning may be needed to achieve optimal performance.
Future Work
Future research directions include optimizing MAVEN's computational efficiency, exploring its application in more complex reasoning tasks, and integrating other reinforcement learning techniques to further enhance performance.
AI Executive Summary
Long-context reasoning is a major challenge for current large language models. Traditional methods often focus only on final answers or static evidence extraction, neglecting feedback on intermediate processes. The MAVEN framework addresses this issue by introducing an editable evidence memory and action-level reward mechanism.
MAVEN was extensively tested on Llama and Qwen models, showing superior performance on LongBench v2, LongReason, and RULER datasets. MAVEN not only improved the accuracy of final answers but also significantly increased evidence sufficiency and reduced distractor retention.
This framework offers a new approach to long-context reasoning by optimizing dynamic evidence navigation rather than one-shot evidence extraction, significantly enhancing the model's reasoning capabilities. Future research will continue to optimize its computational efficiency and explore its application in more complex tasks.
Deep Analysis
Background
Long-context reasoning is a significant challenge for large language models, especially when dealing with large volumes of dispersed information. Recent advancements have increased the context window size, improving models' ability to handle long texts, but effective reasoning remains inadequate. Existing methods often focus on final answers or static evidence extraction, lacking feedback on intermediate processes.
Core Problem
The core problem of long-context reasoning is how to effectively locate, revise, and synthesize evidence dispersed across long texts. Traditional methods often stop exploring after finding locally plausible evidence, leading to incomplete evidence chains.
Innovation
The MAVEN framework introduces an editable evidence memory and action-level reward mechanism, providing more granular process supervision. Its innovation lies in dynamically evaluating changes in evidence state and rewarding actions based on their impact on the evidence state.
Methodology
- �� MAVEN defines an answer-conditioned evidence-state value.
- �� Rewards action-level state transitions: add, link, drop.
- �� Optimizes policy model using GRPO.
- �� Extensively tested on Llama and Qwen models.
Experiments
Experiments were conducted on LongBench v2, LongReason, and RULER datasets using Llama-3.1-8B, Qwen2.5-14B, and Qwen3-30B-A3B models. Evaluation metrics included evidence sufficiency, distractor retention, and final answer accuracy.
Results
MAVEN outperformed existing baselines on all test datasets, particularly in long-context settings. It improved scores on the Llama-3.1-8B model by 3.5 percentage points, significantly increasing evidence sufficiency and reducing distractor retention.
Applications
MAVEN can be applied in various scenarios requiring long text processing, such as legal document analysis, scientific literature review, and complex problem-solving. Its dynamic evidence navigation mechanism provides more efficient reasoning capabilities for these fields.
Limitations & Outlook
MAVEN may face high computational costs when handling extremely long contexts, especially when frequent updates to the evidence memory are required. Future research will continue to optimize its computational efficiency and explore its application in more complex tasks.
Plain Language Accessible to non-experts
Imagine you're in a giant library searching for an answer. Each book is a piece of evidence, and you need to find the right books to answer the question. MAVEN acts like a smart librarian, helping you select, link, and discard books to find the most accurate answer. It rewards you not just for the final books you find but for every correct choice you make along the way.
ELI14 Explained like you're 14
Imagine you're playing a mystery game, trying to find the truth from a pile of clues. MAVEN is like your game assistant, helping you pick the right clues, link related information, and discard misleading ones. It doesn't just care if you find the answer in the end; it rewards you for every smart decision you make during the process.
Glossary
MAVEN (Marginal-Value Evidence Navigation)
A reinforcement learning framework with an editable evidence memory that rewards action-level state transitions.
Used for dynamic evidence navigation in long-context reasoning.
GRPO (Gradient Policy Optimization)
A method for optimizing policy models using gradient information.
Used to optimize MAVEN's policy model.
Evidence-State Value
A function that measures whether the current evidence memory helps predict the correct answer.
Used to evaluate changes in evidence state in MAVEN.
Evidence Synergy
The ability of two evidence pieces to jointly support an answer.
Used to reward link actions in MAVEN.
Marginal Gain
The improvement in the current evidence state due to newly added evidence.
Used to reward add actions in MAVEN.
Open Questions Unanswered questions from this research
- 1 How can MAVEN's performance be further optimized without increasing computational costs?
- 2 How can MAVEN's computational efficiency be improved when handling extremely long contexts?
Applications
Immediate Applications
Legal Document Analysis
MAVEN can help lawyers quickly locate and synthesize relevant legal clauses, improving case analysis efficiency.
Long-term Vision
Scientific Literature Review
MAVEN can assist researchers in extracting and synthesizing relevant findings from a large body of literature, accelerating scientific discovery.
Abstract
Long-context reasoning requires models to locate, revise, and synthesize evidence distributed across lengthy inputs. Existing long-context RL methods usually reward final answers or static evidence extraction, offering little feedback on how intermediate actions change the model's evidence state. We propose Maven, a reinforcement learning framework with an editable evidence memory. Maven defines an answer-conditioned evidence-state value and rewards action-level state transitions: add actions are credited by marginal gain and hindsight contribution, link actions by evidence synergy, and drop actions by improved answer support after removing misleading evidence. These rewards are assigned to the corresponding action spans in GRPO. Across Llama and Qwen models on LongBench v2, LongReason, and RULER, Maven outperforms outcome-only RL and evidence-identification baselines, producing more sufficient evidence sets and lower distractor retention. Our results show that long-context RL benefits from optimizing stateful evidence navigation rather than one-shot evidence extraction.