Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learner
Proposes TP-GRPO with process rewards, achieving +4.32% to +6.67% in Pass@1 on math reasoning models with fewer samples.
Key Findings
Methodology
This paper introduces a thought-level generative process evaluation mechanism driven by intrinsic signals, integrating step merging and capacity-adaptive rewards within the TP-GRPO algorithm. The approach evaluates reasoning steps via internal signals, reducing reliance on explicit problem-solving capabilities of reward models. By aggregating contiguous correct or incorrect steps into 'thoughts', the method structures reward signals more coherently. The capacity-adaptive mechanism dynamically balances exploration and exploitation based on current model proficiency. Experiments on 1.5B and 7B models demonstrate significant performance gains over outcome-only reward baselines, with improvements of +4.32% to +6.67% in Pass@1, using only a fraction of training samples.
Key Results
- On the AIME 2024 benchmark, the 1.5B model trained on 700 problems improved Pass@1 by 4.32%, and with 1800 problems, the gain increased to 5.98%. The 7B model trained on 1070 problems achieved a 6.67% increase, surpassing traditional reward methods.
- The TP-GRPO algorithm, combining process rewards with step merging, effectively mitigates reward hacking and enhances sample efficiency. Ablation studies confirm the importance of thought-level evaluation and adaptive reward tuning.
- Results indicate that structured process rewards enable models to learn reasoning more efficiently, especially in low-resource scenarios, validating the core hypothesis that process-level signals accelerate learning.
Significance
This work addresses the key challenge of sparse reward signals in reinforcement learning for large language models, especially in complex reasoning tasks. By leveraging intermediate reasoning signals and structuring rewards at the thought level, it enhances the learning efficiency and robustness of models. The approach offers a new paradigm for training large-scale reasoning systems, reducing data requirements and improving generalization. It also provides insights into how internal signals can be exploited to guide self-supervised learning, opening avenues for more autonomous and explainable AI systems. The methodology bridges the gap between reinforcement learning and interpretability, with broad implications for natural language understanding, mathematical reasoning, and beyond.
Technical Contribution
The core technical contribution is the development of a thought-level process reward framework that integrates with the TP-GRPO algorithm. This framework employs intrinsic signals for step correctness evaluation, combined with a step merging strategy to form coherent 'thoughts'. The capacity-adaptive reward mechanism dynamically adjusts rewards based on model proficiency, balancing exploration and exploitation. Theoretical analysis proves that the reward design preserves the original optimization objective for correct and incorrect thoughts, mitigating reward hacking. The implementation includes an off-policy training pipeline that improves efficiency and scalability, enabling training on large models with limited data. This work advances the state-of-the-art in reinforcement learning for reasoning tasks by providing a more structured, robust, and efficient training paradigm.
Novelty
This is the first work to integrate thought-level process rewards with a capacity-adaptive mechanism in large reasoning models. Unlike prior methods relying solely on outcome rewards or discriminative PRMs, this approach leverages generative signals for intermediate step evaluation, significantly reducing reliance on the reward model's reasoning ability. The step merging strategy enhances reward interpretability and stability, while the adaptive mechanism ensures balanced exploration. These innovations collectively enable more sample-efficient training and robust reasoning performance, representing a substantial leap forward in reinforcement learning for complex reasoning tasks.
Limitations
- The step-level evaluation relies heavily on prompt design, which can be unstable and sensitive to prompt quality. Improving robustness requires further prompt optimization or fine-tuning of the generative reward model.
- Frequent calls to the generative process increase computational costs, limiting scalability. Future work should explore more efficient evaluation strategies or approximation methods.
- Current error attribution for incorrect solutions may misjudge intermediate steps, especially in complex reasoning chains, potentially affecting reward accuracy and learning stability.
Future Work
Future research will focus on enhancing the stability and robustness of step-level evaluation, possibly through fine-tuning or training dedicated reward models. Developing more efficient sampling and evaluation pipelines will reduce computational overhead. Extending the framework to multi-modal reasoning and more diverse tasks, such as scientific reasoning or multi-step problem solving, will broaden its applicability. Additionally, integrating self-supervised signals and exploring unsupervised reward learning could further improve autonomous reasoning capabilities.
AI Executive Summary
Large-scale reasoning models have demonstrated remarkable potential in tackling complex mathematical problems, yet their training remains hampered by sparse reward signals and inefficient exploration. Traditional reinforcement learning approaches rely solely on outcome-based rewards, which provide limited feedback and often lead to reward hacking and slow convergence. To address these issues, this work introduces a novel framework that leverages process-level rewards at the thought unit level, significantly improving learning efficiency.
The core innovation lies in a thought-level generative process evaluation mechanism driven by intrinsic signals within the model's reasoning trajectories. Instead of requiring the reward model to understand how to solve problems, the method assesses the correctness of individual reasoning steps based on internal cues such as reflection and error signals. These step evaluations are then aggregated into coherent 'thoughts' through a step merging strategy, which enhances reward interpretability and stability. To further refine the learning process, a capacity-adaptive reward mechanism dynamically adjusts the reward strength according to the model's current proficiency, balancing exploration and exploitation.
This integrated approach is implemented within the TP-GRPO algorithm, an extension of the existing GRPO method. The off-policy training pipeline developed ensures efficient utilization of computational resources, enabling large models to be trained on limited data. Extensive experiments on 1.5B and 7B parameter models trained on a fraction of the typical dataset show substantial performance gains: Pass@1 improves by 4.32% to 6.67%, outperforming baseline methods that use only outcome rewards.
These results validate the hypothesis that structured process rewards can accelerate learning in mathematical reasoning tasks. The approach not only reduces data requirements but also enhances model robustness against reward hacking. While challenges remain in evaluation stability and computational costs, this work paves the way for more autonomous, interpretable, and efficient reasoning systems, with broad implications for AI research and practical applications in education, automation, and scientific discovery.
Deep Dive
Plain Language Accessible to non-experts
想象你在学习一道复杂的数学题，传统的方法就像老师只关心你最后的答案是否正确。而这项新方法更像是老师在你解题的每一步都给你反馈，比如你是否正确地理解了问题、每个推理是否合理。老师还会根据你目前的水平，调整反馈的严格程度，鼓励你多尝试不同的解法。这样，你不仅知道最终答案对不对，还能逐步改正每个错误，学得更快更牢。就像在做一道难题时，有个聪明的助手在你每个步骤都帮你检查，告诉你哪里做得好，哪里需要改进。最终，你的解题能力会变得更强，学习也变得更有趣、更高效。
ELI14 Explained like you're 14
想象你在学校解一道超级难的数学题，平时老师只看你最后的答案是不是对的，但其实，知道你每一步怎么想的也很重要。这就像是让你在解题的过程中，有个聪明的朋友在旁边，观察你每个推理步骤，告诉你哪些地方做得好，哪些地方需要改正。这个朋友还能根据你的水平，调整反馈的严格程度，鼓励你多试试不同的方法。这样，你学得更快，也更懂得怎么解复杂的题。虽然这个系统还不是完美，有时候会误判，但它让学习变得更有条理、更有效率，就像有个聪明的老师在你身边，帮你一步步变得更厉害！
Abstract
Large reasoning models (LRMs) have recently shown promise in solving complex math problems when optimized with Reinforcement Learning (RL). But conventional approaches rely on outcome-only rewards that provide sparse feedback, resulting in inefficient optimization process. In this work, we investigate the function of process reward models (PRMs) to accelerate the RL training for LRMs. We propose a novel intrinsic signal-driven generative process evaluation mechanism operating at the thought level to address major bottlenecks in RL-based training. Specifically, instead of requiring PRMs to know how to solve problems, our method uses intrinsic signals in solutions to judge stepwise correctness and aggregate contiguous correct/incorrect steps into coherent 'thought' units. This structured, thought-level rewards enable more reliable credit assignment by reducing ambiguity in step segmentation and alleviating reward hacking. We further introduce a capability-adaptive reward mechanism that dynamically balances exploration and exploitation based on the LRM's current proficiency, guiding learning without stifling creative trial-and-error. These innovations are integrated into a new off-policy RL algorithm, TP-GRPO, which extends grouped proximal optimization with process-based rewards and improves training efficiency. Experiments on 1.5B and 7B parameter LRMs demonstrate that our method achieves higher problem-solving accuracy with significantly fewer training samples than outcome-only reward baselines. The results validate that well-structured process rewards can substantially accelerate LRM optimization in math reasoning tasks. Code is available at https://github.com/cs-holder/tp_grpo.