Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning
Improved long-context reasoning using a data recipe and GRPO setup, achieving average gains of 7.2 points.
Key Findings
Methodology
The paper introduces a data recipe combined with a minimal outcome-based GRPO setup to significantly enhance long-context reasoning. The recipe targets three task categories: retrieval, multi-evidence synthesis, and reasoning. Eight datasets totaling ~14K examples were constructed and tested on three models.
Key Results
- On Qwen3-4B/8B/30B-A3B models, average improvements were +7.2/+3.2/+6.4 points, surpassing existing RL training sets.
- The data recipe also excelled in agent tasks, improving GAIA by 4.8 points and BrowseComp by 7.0 points.
- Compared to DocQA-RL-1.6K and KeyChain-15K, our data recipe performed best across all benchmarks.
Significance
This research significantly enhances long-context reasoning through a data-driven approach, challenging traditional reward engineering methods. It provides new directions for future research and may impact the development of automated agents.
Technical Contribution
Technical contributions include proposing a long-context data recipe without special reward engineering, demonstrating the critical role of data diversity in enhancing reasoning capabilities, and providing new experimental datasets.
Novelty
This is the first to enhance long-context reasoning through a data recipe rather than reward engineering, highlighting the importance of data diversity and task categorization in training.
Limitations
- Dataset scale and diversity are still limited, which may affect model generalization.
- The adaptability of different model architectures to the data recipe was not considered.
Future Work
Future work could explore the construction of more datasets and the adaptability of different model architectures, as well as further enhancing long-context reasoning capabilities.
AI Executive Summary
Long-context reasoning is crucial for modern large language models, especially in automated agents. Existing research mainly focuses on reward engineering, but data diversity remains scarce. This paper significantly enhances long-context reasoning through a data recipe and GRPO setup. Eight datasets were constructed and tested on three models, showing an average improvement of 7.2 points. The method not only excelled in long-context benchmarks but also achieved significant gains in agent tasks. Future research can continue exploring data-driven methods to further enhance the capabilities of automated agents.
Deep Analysis
Background
Long-context reasoning is vital for large language models in automated agents. Existing research mainly focuses on reward engineering, but data diversity remains scarce. This paper proposes a new data-driven method to enhance long-context reasoning capabilities.
Core Problem
Enhancing long-context reasoning capabilities has been a challenge for large language models. Existing methods rely heavily on reward engineering, but insufficient data diversity limits model generalization.
Innovation
The paper proposes a data recipe that enhances long-context reasoning through three task categories: retrieval, multi-evidence synthesis, and reasoning. Unlike traditional reward engineering, this method focuses on data diversity and task categorization.
Methodology
- �� Constructed eight datasets totaling ~14K examples
- �� Used GRPO setup for training
- �� Conducted experiments on three models to evaluate the effectiveness of the data recipe
Experiments
Experiments were conducted on Qwen3-4B/8B/30B-A3B models using seven long-context benchmarks to evaluate the effectiveness of the data recipe. Results showed that the data recipe performed excellently across all benchmarks.
Results
Results showed average improvements of +7.2/+3.2/+6.4 points on three models, surpassing existing RL training sets. The data recipe also excelled in agent tasks, improving GAIA by 4.8 points and BrowseComp by 7.0 points.
Applications
The data recipe can be used to enhance the long-context reasoning capabilities of automated agents, especially in complex tasks requiring extensive information integration.
Limitations & Outlook
Dataset scale and diversity are still limited, which may affect model generalization. Future research could explore the construction of more datasets and the adaptability of different model architectures.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. You need to find all the ingredients from different cabinets and follow the recipe step by step. This method is like a smart assistant that helps you quickly find all the ingredients and tells you how to combine them to make a delicious dish.
ELI14 Explained like you're 14
Hey there! Imagine you're playing a super complex game where you need to find clues from different levels to solve a puzzle. This method is like a super helper that quickly finds all the clues and tells you how to combine them to win the game! Isn't that cool?
Glossary
Reinforcement Learning
A machine learning method that trains models to make decisions through rewards and punishments.
Used to enhance long-context reasoning capabilities.
Data Recipe
A method that enhances model capabilities through data diversity and task categorization.
Used to construct long-context reasoning datasets.
Long-Context Reasoning
The ability of a model to reason over long texts.
The core research subject of the paper.
Group Relative Policy Optimization
An optimization strategy that updates models by comparing advantages across multiple samples.
Used for model training.
GAIA
A benchmark test for evaluating automated agent tasks.
Used to assess the effectiveness of the data recipe.
Open Questions Unanswered questions from this research
- 1 How to further enhance dataset diversity to improve model generalization.
- 2 Research on the adaptability of different model architectures to the data recipe.
Applications
Immediate Applications
Automated Agents
Enhance long-context reasoning capabilities in complex tasks for automated agents.
Long-term Vision
Smart Assistants
Develop smarter assistants capable of reasoning and decision-making in various complex scenarios.
Abstract
Long-context reasoning is an essential capability for large language models, particularly when they are deployed as autonomous agents that must reason over lengthy trajectories. Reinforcement learning (RL) has recently emerged as a dominant paradigm for improving this ability, yet existing work largely focuses on reward engineering while diverse training data remains scarce. We revisit this problem from a data-centric perspective and show that a simple yet effective data recipe alone, paired with a minimal outcome-based GRPO setup, suffices to substantially improve long-context reasoning. Our recipe targets three complementary task families -- retrieval, multi-evidence synthesis, and reasoning -- for which we construct and curate eight datasets totaling ~14K examples. Experiments on three models (Qwen3-4B/8B/30B-A3B) yield average gains of +7.2/+3.2/+6.4 points across seven long-context benchmarks, surpassing prior RL training sets. We further demonstrate that these gains transfer to agentic tasks, where continuing RL training on an agent-tuned model with our data recipe improves GAIA by +4.8 and BrowseComp by +7.0 points. We will release our datasets to facilitate future research.