GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning
GenPRM scales test-time compute via generative reasoning, outperforming GPT-4o on ProcessBench.
Key Findings
Methodology
GenPRM redefines process supervision as a generative task using Chain-of-Thought reasoning and code verification, employing Relative Progress Estimation for high-quality labels.
Key Results
- GenPRM surpasses GPT-4o on ProcessBench with only 23K training data; 1.5B model outperforms GPT-4o, 7B model exceeds Qwen2.5-Math-PRM-72B.
- GenPRM excels in mathematical reasoning tasks, especially on complex problems.
- GenPRM demonstrates strong critique capabilities, significantly enhancing policy model performance.
Significance
GenPRM establishes a new paradigm for process supervision by combining generative reasoning and code verification, addressing limitations in existing PRMs' supervision and scaling capabilities, with significant impact on academia and industry.
Technical Contribution
GenPRM fundamentally redefines process supervision through generative reasoning and code verification, offering new theoretical guarantees and engineering possibilities, significantly enhancing PRM scaling capabilities.
Novelty
GenPRM is the first to combine generative reasoning with code verification, significantly improving process supervision model performance and scaling capabilities, offering fundamental innovation over traditional PRMs.
Limitations
- GenPRM may still face challenges in extremely complex reasoning tasks, requiring further optimization.
- The relatively small amount of training data may affect generalization capabilities.
Future Work
Future work could explore applications on larger datasets, optimize the integration of generative reasoning and code verification, and enhance model generalization and efficiency.
AI Executive Summary
Recent advances in large language models (LLMs) have shown the potential of using process reward models (PRMs) as verifiers to enhance LLM performance. However, existing PRMs face limitations in process supervision and scaling capabilities. GenPRM redefines process supervision through generative reasoning and code verification, significantly improving model performance. In ProcessBench and mathematical reasoning tasks, GenPRM excels, particularly on complex problems. Through test-time scaling, GenPRM's 1.5B model surpasses GPT-4o, and the 7B model exceeds Qwen2.5-Math-PRM-72B. GenPRM establishes a new paradigm for process supervision by combining generative reasoning and code verification, addressing limitations in existing PRMs' supervision and scaling capabilities, with significant impact on academia and industry. Future work could explore applications on larger datasets, optimize the integration of generative reasoning and code verification, and enhance model generalization and efficiency.
Deep Analysis
Background
Recent developments in large language models (LLMs) have highlighted the potential of process reward models (PRMs) as verifiers. However, existing PRMs struggle with limited process supervision and scaling capabilities, making them less effective across diverse tasks and models.
Core Problem
Current PRMs rely on scalar value predictions without leveraging LLMs' generative capabilities, limiting process supervision and scaling abilities, and hindering performance on complex tasks.
Innovation
GenPRM redefines process supervision through generative reasoning and code verification, viewing it as a generative task rather than a discriminative one, significantly enhancing model performance and scaling capabilities.
Methodology
- �� Generative Reasoning: Redefines process supervision with explicit Chain-of-Thought (CoT) reasoning and code verification.
- �� Relative Progress Estimation (RPE): Uses a relative criterion for label estimation, improving label accuracy.
- �� Rationale Synthesis Framework: Combines code verification to obtain high-quality process supervision reasoning data.
Experiments
Experiments were conducted on ProcessBench and multiple mathematical reasoning tasks using 23K training data from the MATH dataset. Through test-time scaling, GenPRM surpasses existing PRMs across multiple benchmarks.
Results
GenPRM surpasses GPT-4o on ProcessBench with only 23K training data; 1.5B model outperforms GPT-4o, 7B model exceeds Qwen2.5-Math-PRM-72B.
Applications
GenPRM can enhance LLM performance in complex reasoning tasks, particularly in mathematical reasoning and policy model refinement.
Limitations & Outlook
GenPRM may still face challenges in extremely complex reasoning tasks, requiring further optimization. The relatively small amount of training data may affect generalization capabilities.
Plain Language Accessible to non-experts
Imagine a factory where workers need to complete a series of complex tasks. GenPRM acts like a smart supervisor, not only assessing task completion but also analyzing each worker's steps to provide improvement suggestions. This approach greatly enhances factory efficiency, enabling workers to complete tasks more effectively. This supervisor doesn't just rely on simple scoring systems but observes and analyzes each step to ensure smooth operation.
ELI14 Explained like you're 14
Imagine you're playing a complex game, and GenPRM is like a super helper. It not only tells you your current score but also analyzes each of your moves, offering tips to help you level up faster. It's like having a smart friend by your side, giving you hints to perform better in the game. Isn't that cool?
Glossary
Generative Reasoning
The process of reasoning and judgment using generative models, combining generative and reasoning capabilities.
Used to redefine process supervision and enhance model performance.
Process Reward Model
A model used to verify and enhance LLM performance through a reward mechanism guiding model optimization.
Acts as a verifier to improve LLM performance in complex tasks.
Code Verification
The process of generating and executing code to verify the correctness of reasoning steps, ensuring the accuracy of the reasoning process.
Combined with generative reasoning to improve process supervision accuracy.
Relative Progress Estimation
Estimates label accuracy by comparing the progress of current and next states, improving label reliability.
Used to obtain high-quality process supervision labels.
Chain-of-Thought
Explicit reasoning chains used for reasoning and judgment, enhancing model reasoning capabilities.
Combined with code verification to enhance generative reasoning effectiveness.
Open Questions Unanswered questions from this research
- 1 How can GenPRM be applied to larger datasets to improve generalization capabilities?
- 2 How can the integration of generative reasoning and code verification be optimized to enhance efficiency?
Applications
Immediate Applications
Mathematical Reasoning Tasks
GenPRM can enhance performance in mathematical reasoning tasks, especially on complex problems.
Long-term Vision
General Artificial Intelligence
Optimizing generative reasoning and code verification, GenPRM has the potential to advance general artificial intelligence.
Abstract
Recent advancements in Large Language Models (LLMs) have shown that it is promising to utilize Process Reward Models (PRMs) as verifiers to enhance the performance of LLMs. However, current PRMs face three key challenges: (1) limited process supervision and generalization capabilities, (2) dependence on scalar value prediction without leveraging the generative abilities of LLMs, and (3) inability to scale the test-time compute of PRMs. In this work, we introduce GenPRM, a generative process reward model that performs explicit Chain-of-Thought (CoT) reasoning with code verification before providing judgment for each reasoning step. To obtain high-quality process supervision labels and rationale data, we propose Relative Progress Estimation (RPE) and a rationale synthesis framework that incorporates code verification. Experimental results on ProcessBench and several mathematical reasoning tasks show that GenPRM significantly outperforms prior PRMs with only 23K training data from MATH dataset. Through test-time scaling, a 1.5B GenPRM outperforms GPT-4o, and a 7B GenPRM surpasses Qwen2.5-Math-PRM-72B on ProcessBench. Additionally, GenPRM demonstrates strong abilities to serve as a critic model for policy model refinement. This work establishes a new paradigm for process supervision that bridges the gap between PRMs and critic models in LLMs. Our code, model, and data will be available in https://ryanliu112.github.io/GenPRM.