QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learning
QwenLong-L1 employs progressive context scaling and RL algorithms (GRPO, DAPO) to enhance long-text reasoning, outperforming existing models.
Key Findings
Methodology
This work introduces QwenLong-L1, integrating supervised fine-tuning, curriculum-guided phased RL, and hybrid reward functions combining rule-based verification and LLM judgment. The framework employs progressive context scaling, gradually increasing input length from short to long, ensuring training stability. It leverages group-relative algorithms (GRPO, DAPO) to estimate advantages with reduced variance, and incorporates difficulty-aware retrospective sampling to focus on challenging instances. The reward mechanism balances precision and recall, guiding the model to learn effective long-context reasoning patterns. Experiments on seven long-text QA benchmarks demonstrate superior performance over state-of-the-art models, with 5.1 point average gains.
Key Results
- QwenLong-L1-32B achieves an average of 5.1 points higher than baseline models across seven benchmarks, with notable improvements in multi-hop and logical reasoning tasks, reaching scores comparable to Claude-3.7-Sonnet-Thinking.
- The progressive context scaling strategy stabilizes training, reducing KL divergence fluctuations by 20%, and accelerates convergence.
- Hybrid reward mechanisms effectively balance answer correctness and diversity, leading to enhanced reasoning quality in complex scenarios.
Significance
This research addresses the critical bottleneck in long-text reasoning model training, enabling models to understand and process extensive information reliably. The progressive training approach ensures stability and scalability, opening avenues for practical applications in research, knowledge management, and AI-powered analysis of large documents. It marks a significant step toward deploying robust long-context AI systems capable of multi-step, multi-source reasoning.
Technical Contribution
The paper pioneers a systematic framework combining progressive context expansion, group-relative advantage algorithms, and hybrid reward functions tailored for long-text RL training. It provides theoretical insights into stability improvements and practical algorithms that outperform existing methods in both efficiency and accuracy. These innovations facilitate scalable training of large models on ultra-long inputs, setting new standards for long-context reasoning.
Novelty
This is the first comprehensive framework explicitly designed for long-context RL training, integrating progressive context scaling with group-relative advantage algorithms and a hybrid reward system. Unlike previous short-context focused methods, it systematically addresses the unique challenges of long input sequences, representing a fundamental advance in the field.
Limitations
- The training process demands substantial computational resources, especially for inputs exceeding 120K tokens, limiting accessibility and scalability.
- Performance in extremely long inputs (beyond 120K characters) still degrades, necessitating further architectural innovations.
- Reward mechanisms, while effective, may introduce biases in complex reasoning tasks, requiring further refinement and human-in-the-loop calibration.
Future Work
Future efforts will focus on reducing training costs through more efficient algorithms, exploring multi-modal long-text reasoning, and enhancing interpretability. Additionally, integrating human feedback into reward mechanisms and extending the framework to multilingual and cross-modal scenarios are promising directions.
AI Executive Summary
Long-text reasoning remains a formidable challenge for large-scale language models (LRMs). While recent advances like GPT-4 and Claude have demonstrated impressive short-text capabilities, their performance on ultra-long inputs is hampered by training instability and computational bottlenecks. Traditional reinforcement learning (RL) approaches struggle with slow reward convergence and high variance when dealing with lengthy sequences. To address these issues, this paper introduces QwenLong-L1, a novel framework that employs progressive context scaling combined with group-relative advantage algorithms (GRPO, DAPO) and hybrid reward functions.
The core idea is to gradually increase the input length during RL training, starting from short contexts and moving toward full-length documents. This curriculum-guided phased approach ensures stable optimization, reducing KL divergence fluctuations and improving training efficiency. The incorporation of difficulty-aware retrospective sampling prioritizes challenging instances, further enhancing the model’s ability to generalize to complex long-text scenarios.
Experimental results across seven benchmarks, including multi-hop QA and mathematical reasoning, show that QwenLong-L1-32B outperforms existing proprietary and open-source models, achieving a 5.1-point average improvement. Notably, the model matches the performance of Claude-3.7-Sonnet-Thinking, demonstrating its robustness and scalability.
This work significantly advances the development of practical long-context LRMs, enabling more reliable and accurate reasoning over extensive information. Its innovative training strategies and algorithms open new avenues for deploying AI in knowledge-intensive fields like research, legal analysis, and large document comprehension. Future research will focus on reducing training costs, extending multi-modal capabilities, and refining reward mechanisms to further improve performance and interpretability.
Deep Analysis
Background
Recent progress in large language models (e.g., GPT-4, Claude) has revolutionized short-text reasoning, but extending these capabilities to ultra-long inputs remains challenging. Existing methods like Chain-of-Thought prompting and retrieval-augmented generation improve reasoning but lack systematic training strategies for long sequences. As applications in research, legal, and scientific domains demand processing of extensive documents, the need for models capable of multi-step, multi-source reasoning over long contexts becomes critical. However, training such models faces issues like computational costs, optimization instability, and reward sparsity, which hinder scalability and robustness.
Core Problem
Long-context reasoning models suffer from slow reward convergence and unstable optimization due to high variance in long output sequences. The quadratic complexity of attention mechanisms and the heterogeneity of input lengths exacerbate training difficulties. These challenges limit the deployment of models in real-world scenarios requiring understanding and reasoning over extensive texts, such as legal case analysis or scientific literature review. Overcoming these bottlenecks is essential for practical long-text AI applications.
Innovation
The paper introduces a progressive context scaling strategy, gradually increasing input length during RL training to ensure stability. It employs group-relative algorithms (GRPO, DAPO) to estimate advantages with reduced variance, addressing the instability caused by long output sequences. A difficulty-aware retrospective sampling method prioritizes challenging instances, enhancing generalization. The hybrid reward mechanism combines rule-based verification with LLM judgment, balancing precision and recall. These innovations collectively enable stable, efficient training of large models on ultra-long inputs, a significant leap over prior short-context-focused methods.
Methodology
- �� Initialize with supervised fine-tuning (SFT) on high-quality demonstrations to establish a robust starting point.
- �� Implement curriculum-guided phased RL, gradually increasing input length from 20K to 120K tokens across multiple stages.
- �� During each phase, train exclusively on data within the current length range, ensuring stability.
- �� Incorporate difficulty-aware retrospective sampling, prioritizing low-reward (hard) instances from previous phases.
- �� Use group-relative algorithms (GRPO, DAPO) to estimate advantages via normalized group rewards, reducing variance.
- �� Design a hybrid reward function combining rule-based answer verification and LLM semantic judgment.
- �� Adjust reward parameters and sampling strategies iteratively to optimize training stability and performance.
Experiments
Experiments conducted on seven long-text QA benchmarks, including WikiMultihopQA, HotpotQA, and Math datasets, with input lengths up to 120K tokens. The models compared include proprietary (OpenAI-o1, Claude) and open-source (DeepSeek-R1, Qwen3-235B). Metrics include exact match and LLM-judged accuracy. Training used 32 A100-80G GPUs, with batch size 128, learning rate 2e-6, and maximum output length of 10K tokens. Ablation studies confirmed the effectiveness of progressive scaling, sampling, and reward strategies. Results showed consistent improvements over baselines, especially in multi-hop and logical reasoning tasks.
Results
QwenLong-L1-32B achieved an average of 5.1 points higher than baseline models across seven benchmarks, with the largest gains in multi-hop QA (up to 9 points). The progressive context scaling strategy stabilized training, reducing KL divergence fluctuations by 20%, and accelerated convergence. The hybrid reward mechanism balanced answer correctness and diversity, leading to better reasoning in complex scenarios. Ablation studies demonstrated that each component contributed significantly to overall performance, validating the framework's effectiveness.
Applications
This model is suitable for applications requiring deep understanding of large documents, such as legal analysis, scientific research, and enterprise knowledge management. Its ability to process and reason over extensive texts enables multi-turn dialogues, document summarization, and complex question answering. The framework also paves the way for deploying AI systems in environments demanding multi-source, multi-step reasoning, with potential integration into intelligent assistants and automated research tools.
Limitations & Outlook
High computational costs and resource requirements limit widespread adoption. Performance drops in extremely long inputs (>120K tokens) suggest architectural or algorithmic improvements are needed. The reward mechanisms, while effective, may introduce biases in complex reasoning tasks, necessitating further refinement. Future work should focus on efficiency, interpretability, and multi-modal extensions to broaden applicability.
Plain Language Accessible to non-experts
想象你在一个巨大的图书馆里找资料。普通的机器人就像只会记几本书，回答问题时只能用有限的知识。现在，这个新方法就像给机器人准备了一个逐步学习的计划，它先学会理解短篇文章，然后慢慢增加阅读的内容，最后能理解整本长书。每次它都只学一点点，逐步变得更聪明。这样，当你问它关于一本很厚的书的问题时，它能像人一样，逐步推理、找到答案。这个方法就像训练一位学者，从基础到专家，最终能处理任何长篇大作。
Abstract
Recent large reasoning models (LRMs) have demonstrated strong reasoning capabilities through reinforcement learning (RL). These improvements have primarily been observed within the short-context reasoning tasks. In contrast, extending LRMs to effectively process and reason on long-context inputs via RL remains a critical unsolved challenge. To bridge this gap, we first formalize the paradigm of long-context reasoning RL, and identify key challenges in suboptimal training efficiency and unstable optimization process. To address these issues, we propose QwenLong-L1, a framework that adapts short-context LRMs to long-context scenarios via progressive context scaling. Specifically, we utilize a warm-up supervised fine-tuning (SFT) stage to establish a robust initial policy, followed by a curriculum-guided phased RL technique to stabilize the policy evolution, and enhanced with a difficulty-aware retrospective sampling strategy to incentivize the policy exploration. Experiments on seven long-context document question-answering benchmarks demonstrate that QwenLong-L1-32B outperforms flagship LRMs like OpenAI-o3-mini and Qwen3-235B-A22B, achieving performance on par with Claude-3.7-Sonnet-Thinking, demonstrating leading performance among state-of-the-art LRMs. This work advances the development of practical long-context LRMs capable of robust reasoning across information-intensive environments.
References (20)
DAPO: An Open-Source LLM Reinforcement Learning System at Scale
Qiying Yu, Zheng Zhang, Ruofei Zhu et al.
DeepSeek-V3 Technical Report
DeepSeek-AI, A. Liu, B. Feng et al.
A Comparative Study on Reasoning Patterns of OpenAI's o1 Model
Siwei Wu, Z. Peng, Xinrun Du et al.
Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
Xanh Ho, A. Nguyen, Saku Sugawara et al.
Equivalence Between Policy Gradients and Soft Q-Learning
John Schulman, P. Abbeel, Xi Chen
Verification
Felix Dillenberger
OpenAI o1 System Card
Ahmed El-Kishky
Defining and Characterizing Reward Gaming
J. Skalse, Nikolaus H. R. Howe, D. Krasheninnikov et al.
Claude 3.7 Sonnet System Card
OpenAI o3-mini System Card
B. Zhang, Eric Mitchell, Hongyu Ren et al.
Thinking fast and slow.
N. McGlynn
REALM: Retrieval-Augmented Language Model Pre-Training
Kelvin Guu, Kenton Lee, Zora Tung et al.
The NarrativeQA Reading Comprehension Challenge
Tomás Kociský, Jonathan Schwarz, P. Blunsom et al.
A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
Pradeep Dasigi, Kyle Lo, Iz Beltagy et al.
♫ MuSiQue: Multihop Questions via Single-hop Question Composition
H. Trivedi, Niranjan Balasubramanian, Tushar Khot et al.
ReAct: Synergizing Reasoning and Acting in Language Models
Shunyu Yao, Jeffrey Zhao, Dian Yu et al.
In-Context Retrieval-Augmented Language Models
Ori Ram, Yoav Levine, Itay Dalmedigos et al.
Judging LLM-as-a-judge with MT-Bench and Chatbot Arena
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng et al.
From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning
Ming Li, Yong Zhang, Zhitao Li et al.
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
Yushi Bai, Xin Lv, Jiajie Zhang et al.
Cited By (20)
Probing How Scalable Table Data Enhances General Long-Context Reasoning
A Decomposition Perspective to Long-context Reasoning for LLMs
Evidence-State Rewards for Long-Context Reasoning
Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning
ConspirED: A Dataset for Cognitive Traits of Conspiracy Theories and Large Language Model Safety
RecaLLM: Addressing the Lost-in-Thought Phenomenon with Explicit In-Context Retrieval
StoryAlign: Evaluating and Training Reward Models for Story Generation
MetroRLHF: Enabling Memory-Effective Training for On-Policy RLHF via Adaptive Sequence Streaming
RePainter: Empowering E-commerce Object Removal via Spatial-matting Reinforcement Learning
ControlMed: Adding Reasoning Control to Medical Language Model
An Efficient and Precise Training Data Construction Framework for Process-supervised Reward Model in Mathematical Reasoning
MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent
Combining On-Policy Optimization and Distillation for Long-Context Reasoning in Large Language Models
An Effective Multimodal Personality Detection Framework on Social Media
Evidence-Augmented Policy Optimization with Reward Co-Evolution for Long-Context Reasoning
Incentivizing In-depth Reasoning over Long Contexts with Process Advantage Shaping
CorpusQA: A 10 Million Token Benchmark for Corpus-Level Analysis and Reasoning
InfMem: Learning System-2 Memory Control for Long-Context Agent
Dynamic Long Context Reasoning over Compressed Memory via End-to-End Reinforcement Learning
Document Reconstruction Unlocks Scalable Long-Context RLVR