QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation
QuestA introduces partial solutions during RL training, boosting 1.5B models' math reasoning by over 10% on benchmarks.
Key Findings
Methodology
This paper proposes QuestA, which incorporates partial solutions as prompts during RL training to ease problem difficulty and densify reward signals. Using high-difficulty problem filtering from OpenR1-220K, the approach employs a curriculum strategy with decreasing hint ratios (p=50% to 25%). It combines the GRPO algorithm with question augmentation, guiding models through intermediate reasoning steps. The core is to scaffold complex problems, enabling models to learn deep reasoning pathways efficiently. Experiments demonstrate that QuestA achieves new SOTA on math benchmarks (AIME24, AIME25, HMMT25) with over 10% improvements, significantly outperforming baseline RL methods.
Key Results
- QuestA reaches 72.50% pass@1 on AIME24 (+10.73%), 62.29% on AIME25 (+12.79%), and 41.67% on HMMT25 (+10.11%), surpassing previous models. It maintains higher pass@k across the spectrum, especially on challenging problems, indicating improved reasoning depth.
- Training dynamics show faster convergence, stable entropy, and richer output diversity. Ablation studies confirm that decreasing hint ratios enhances efficiency without performance loss. The method's compatibility with various RL algorithms broadens its applicability.
- The results validate that partial-solution scaffolding effectively addresses sparse reward issues, enabling models to explore complex reasoning trajectories more reliably, thus pushing the limits of current RL-based reasoning training.
Significance
This work addresses fundamental bottlenecks in scaling reasoning capabilities of large models via RL. By systematically scaffolding problems with partial solutions, it overcomes sparse reward and exploration issues, leading to substantial performance gains in high-complexity tasks. The approach bridges the gap between supervised fine-tuning and reinforcement learning, offering a scalable pathway for advancing AI reasoning. Its success on math benchmarks suggests broad applicability to scientific, logical, and programming domains, promising transformative impacts in automated reasoning and problem-solving AI systems.
Technical Contribution
The key technical innovation is the integration of partial solutions as prompts, combined with a curriculum strategy that gradually reduces hint ratios. Theoretically, the paper formalizes how question augmentation reshapes the learnability landscape, providing bounds on sample efficiency and convergence guarantees. Practically, the method is compatible with existing RL frameworks, and the open-source pipeline facilitates replication and extension. Theoretical analysis demonstrates that partial-solution hints improve the probability of sampling successful trajectories, thus accelerating training and enhancing generalization.
Novelty
This is the first systematic application of partial-solution question augmentation within RL training for reasoning tasks. Unlike prior work focusing solely on reward shaping or supervised fine-tuning, QuestA leverages input-level scaffolding to guide exploration. Its curriculum-based hint reduction strategy and theoretical validation distinguish it from existing approaches, marking a significant step forward in addressing sparse reward challenges in complex reasoning models.
Limitations
- Dependence on high-quality partial solutions limits applicability where such data is scarce. The method's effectiveness on extremely long or multi-step problems remains to be validated. Computational costs increase with multiple sampling and filtering stages.
- Parameter tuning for hint ratios and filtering thresholds is somewhat manual, requiring further automation. The approach's scalability to larger models and multi-modal data needs exploration. Additionally, the method assumes availability of detailed solution trajectories, which may not always be feasible.
Future Work
Future directions include automating partial solution generation, integrating multi-modal inputs, and extending the framework to broader reasoning domains such as scientific discovery and legal reasoning. Developing adaptive curriculum strategies and reducing data annotation costs will further enhance scalability. Exploring larger models and multi-task learning paradigms will also be key to pushing reasoning capabilities to new heights.
AI Executive Summary
Large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation, yet their reasoning abilities, especially on complex mathematical and logical tasks, remain limited. Reinforcement learning (RL) has been adopted to improve these capabilities, but traditional RL faces significant challenges, notably sparse rewards and slow convergence on high-difficulty problems. This paper introduces QuestA, a novel question augmentation framework that incorporates partial solutions as prompts during RL training, effectively scaffolding the learning process.
By filtering the most challenging problems from datasets like OpenR1-220K and employing a curriculum strategy that gradually reduces hint ratios, QuestA guides models through intermediate reasoning steps. This approach densifies reward signals, accelerates training, and enhances the model’s ability to explore complex solution trajectories. The core innovation lies in decomposing difficult problems into manageable sub-tasks, enabling models to learn deep reasoning pathways more efficiently.
Experimental results on benchmarks such as AIME24, AIME25, and HMMT25 show that QuestA achieves state-of-the-art performance with over 10% improvements in pass@1 metrics for 1.5B parameter models. The method not only boosts accuracy but also maintains solution diversity and robustness across different k-values, outperforming baseline RL methods and even larger models.
Theoretical analysis confirms that question augmentation reshapes the learnability landscape, increasing the likelihood of sampling successful trajectories and reducing training sample complexity. The approach is compatible with existing RL algorithms, easy to integrate, and open-sourced, making it accessible for widespread adoption.
Overall, QuestA offers a significant step forward in scalable reasoning training, with broad implications for AI systems in scientific research, automated reasoning, and complex problem-solving. Future work will focus on automating partial solution generation, extending to multi-modal data, and applying to diverse reasoning domains, aiming to realize AI’s full potential in understanding and solving the world’s most challenging problems.
Deep Analysis
Background
近年来，随着GPT、PaLM等大规模预训练模型的出现，推理能力得到了显著提升，但在高难度任务中仍受限于训练数据的多样性和奖励稀疏问题。传统RL方法如GRPO、DAPO在强化推理方面取得一定进展，但在复杂问题上收敛缓慢，效果有限。Supervised Fine-Tuning（SFT）虽能改善部分性能，但难以突破推理深度瓶颈。近年来，问题增强和多阶段训练策略逐渐成为研究热点，旨在通过结构化提示提升模型推理能力。
Core Problem
核心问题在于如何在强化学习中有效扩展模型的推理深度，尤其在面对高难度、稀疏奖励的任务时。现有方法多依赖硬题训练，导致训练效率低下，模型容易陷入局部最优。难题筛选和提示设计缺乏系统性，难以实现高效学习。如何在保证训练速度的同时，提升模型的推理深度，成为亟待解决的关键问题。
Innovation
本文提出QuestA，创新点包括：1）引入部分解答作为提示，逐步引导模型学习复杂推理路径；2）设计多阶段递减提示比例的课程策略，平衡训练难度与效率；3）结合理论分析，证明部分解答增强能提升样本效率和探索能力。这些创新有效缓解稀疏奖励问题，推动模型在高难度推理任务中的表现突破。
Methodology
- �� 选择高难度问题集（OpenR1-220K）筛选最难样本。• 利用深度推理模型Nemotron-1.5B采样多轮答案，筛除易题。• 设计逐步递减提示比例（p=50%到25%），在RL训练中引入部分解答。• 采用GRPO算法，结合问题增强策略，优化样本利用率。• 通过多轮筛选确保模型在硬题上的学习效果。• 理论分析部分，定义解集与模型容量集，证明部分解答增强能显著提升样本效率。
Experiments
在OpenR1-220K数据集基础上，结合Nemotron-1.5B和DeepScaleR-1.5B模型，进行多轮强化学习训练。采用pass@k指标评估模型推理能力，比较QuestA与传统RL方法的性能差异。通过不同提示比例和筛选策略，验证方法的有效性。实验还包括消融分析，探讨逐步递减策略对训练速度和性能的影响。所有训练在高性能GPU集群上完成，确保结果的可靠性。
Results
QuestA在AIME24、AIME25、HMMT25等数学基准上均实现了SOTA，提升幅度超过10%。在pass@1指标上，QuestA达72.50%、62.29%、41.67%，远优于未增强模型。训练动态显示，QuestA模型收敛更快，奖励密集，避免entropy崩溃。消融实验表明，逐步递减提示比例能进一步优化训练效率，显著缩短收敛时间。整体结果验证了部分解答引导在硬推理任务中的有效性。
Applications
该方法适用于需要高深推理能力的自动问答、数学证明、逻辑推理等场景。可结合现有RL框架，提升模型在复杂任务中的表现。未来可扩展到科学计算、自动编程等领域，推动AI在专业领域的应用落地。通过优化训练流程，降低训练成本，提升模型的实用性和普适性。
Limitations & Outlook
目前依赖高质量的部分解答数据，数据采集成本较高。模型在极端复杂或长尾问题上的泛化能力仍有待验证。训练参数调节较为依赖经验，缺乏自适应机制。未来需结合自动生成技术，降低数据依赖，提升鲁棒性。模型规模限制也影响推理深度，需结合更大模型或多模态信息进行突破。
Plain Language Accessible to non-experts
想象你在学习一项复杂的手工艺，比如拼装一个复杂的模型。直接从零开始拼装很难，容易出错，也很慢。于是，你先看一些部分的示范，学习每个步骤的技巧，然后再自己动手。QuestA的方法也是这样，它在训练模型时，先给出问题的部分解答，让模型学习每个小步骤。这样，模型就像有了“指南针”，逐步掌握复杂推理的技巧。通过不断练习，模型变得越来越擅长解决难题，就像你逐渐变成拼装高手一样。这种逐步学习的方法，让模型在面对高难度问题时，能更快、更好地找到答案，就像你在拼装模型时，先学会拼装基础部分，再逐步完成整个作品一样。
ELI14 Explained like you're 14
想象你在玩一个超级难的拼图游戏，一开始看着一堆乱七八糟的碎片，不知道从哪里开始。于是，你的朋友告诉你一些线索，比如“这个边缘的碎片在左边”，或者“这个颜色的碎片可以拼在一起”。这样，你就可以先拼出边框，慢慢填充里面的内容。QuestA也是这样，它在训练模型时，先给模型一些“线索”——就是问题的部分答案，让模型学会一步步解决复杂的难题。这样，模型就像变得更聪明了，能更快找到正确答案。就像你用线索拼拼图一样，模型用部分答案拼出完整的推理过程。最终，它变得非常擅长解决那些以前觉得太难的问题，就像拼图高手一样厉害！
Abstract
Reinforcement learning (RL) has emerged as a central paradigm for training large language models (LLMs) in reasoning tasks. Yet recent studies question RL's ability to incentivize reasoning capacity beyond the base model. This raises a key challenge: how can RL be adapted to solve harder reasoning problems more effectively? To address this challenge, we propose a simple yet effective strategy via Question Augmentation: introduce partial solutions during training to reduce problem difficulty and provide more informative learning signals. Our method, QuestA, when applied during RL training on math reasoning tasks, not only improves pass@1 but also pass@k-particularly on problems where standard RL struggles to make progress. This enables continual improvement over strong open-source models such as DeepScaleR and OpenMath Nemotron, further enhancing their reasoning capabilities. We achieve new state-of-the-art results on math benchmarks using 1.5B-parameter models: 72.50% (+10.73%) on AIME24, 62.29% (+12.79%) on AIME25, and 41.67% (+10.11%) on HMMT25. Code, data and model are available at https://github.com/foreverlasting1202/QuestA.