Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning
CRM links step rewards to outcomes via conditional hazards, reaching 43.3% on AIME24 in verifier-free RL, 16.7 points above PURE.
Key Findings
Methodology
CRM models reasoning as a finite-horizon MDP and predicts h(t)=Pr(z=t|z≥t), the conditional probability that step t is the first irreversible error. The survival probability S(T)=∏t(1−h(t)) represents eventual correctness. Potential-Based Reward Shaping yields rt=log(1−h(t)); losses LS, LW, and Lz train correct trajectories, failed outcomes, and first-error locations.
Key Results
- In Best-of-N, CRM reaches 56.6% on MATH500 at N=32 with Qwen2.5-3B-Instruct, 1.4 points above the strongest baseline; with LLaMA3.1-8B it reaches 69.1% on GSM-Plus at N=32.
- In beam search, CRM obtains 64.07% on MATH500 at N=100 with Qwen2.5-Math-7B, versus 60.73% for ORM, and 48.40% on the OOD Gaokao2023 benchmark.
- For verifier-free RL, CRM reaches 43.3% on AIME24, outperforming PURE by 16.7 points. It leads or nearly leads on MATH500, Minerva Math, OlympiadBench, AIME25, and AMC23.
Significance
The work addresses a central weakness of process supervision: local scores may not reflect whether a complete reasoning trajectory succeeds. By giving every reward a probability-based meaning, CRM supports both within-question selection and global cross-question ranking. This is important for reasoning tasks where ground-truth verifiers are expensive, unavailable, or difficult to scale, and it offers a more stable alternative to heuristic dense rewards during search and RL.
Technical Contribution
CRM differs from independent step classification and local Q-value ranking by modeling a conditional failure process. The chain rule decomposes final success into ∏(1−h(t)), explicitly tying every step to the outcome. Failed trajectories receive additional supervision at the first wrong step through p(zi). The combined LS, LW, and Lz objectives preserve probabilistic semantics while PBRS supplies dense, policy-preserving rewards.
Novelty
The fundamental novelty is not merely adding outcome labels to a PRM, but constructing a coherent probability model connecting prefix-conditioned step risk, first failure, and final correctness. Unlike vanilla PRM, PQM, and IPRM, CRM provides a common scale for comparing partial and complete trajectories across samples while retaining temporal dependence.
Limitations
- Training uses Math-Shepherd-style process labels and first-error indices, which are costly, subjective, or unavailable for open-ended domains such as science and tool use.
- Experiments focus on mathematical reasoning and Qwen/LLaMA backbones; generalization to code, multimodal tasks, and long-horizon interaction is untested.
- The model assumes that entering a wrong state is effectively irreversible, whereas real reasoning can sometimes backtrack, repair an error, or use external tools.
Future Work
Future work should model recoverable errors, hierarchical steps, tool calls, and explicit backtracking. Important directions include automatic first-error discovery, calibration across domains, parameter-efficient training, larger-model evaluation, and cost-aware online RL. Broader benchmarks are needed to test whether CRM remains robust when correctness cannot be cleanly reduced to a binary label.
AI Executive Summary
Large language models increasingly solve problems through explicit multi-step reasoning. Yet outcome reward models score only the final answer, while vanilla process reward models often classify each step independently. PQM ranks neighboring steps and IPRM parameterizes outcome rewards from process scores, but neither fully explains how a particular prefix causes success or failure. Such ambiguity can produce reward hacking: the learned score rises while task accuracy falls.
Zhang and colleagues propose Conditional Reward Modeling, or CRM. The model treats reasoning as a temporal process that may first enter an irreversible wrong state at step z. It estimates a conditional hazard h(t), then uses the probability chain rule to define the survival probability S(T)=∏(1−h(t)). Potential-Based Reward Shaping converts this quantity into dense rewards, rt=log(1−h(t)). Correct trajectories optimize LS; failed trajectories optimize LW and the probability of their first wrong step, Lz.
Across Best-of-N, beam search, and RLOO reinforcement learning, CRM consistently improves selection and optimization. With Qwen2.5-3B-Instruct, it reaches 56.6% on MATH500 at N=32; with Qwen2.5-Math-7B, beam search reaches 64.07% on MATH500 at N=100. In verifier-free RL, CRM obtains 43.3% on AIME24, 16.7 points above PURE. The approach is promising because it supplies outcome-aligned, comparable rewards without requiring ground-truth verification, although its reliance on mathematical process labels and an irreversible-error assumption limits current generality.
Deep Analysis
Background
Reasoning research has progressed from chain-of-thought prompting to Best-of-N, beam search, and RL. DeepSeek-R1 demonstrates the power of verifiable rewards, but answer checking does not scale easily to open-ended tasks. ORMs give only terminal feedback; PRMs such as Math-Shepherd provide step-level supervision. PQM and IPRM improve structure, yet leave temporal causality and process–outcome alignment incomplete.
Core Problem
For trajectory y=(a1,…,aT), a reward model must estimate whether each partial prefix leads toward a correct answer. Independent step scores ignore prior context; local rankings do not define a shared cross-sample scale; terminal labels distribute credit ambiguously. These weaknesses make search brittle and allow reward hacking during RL.
Innovation
CRM introduces the first wrong-state index z and predicts h(t)=Pr(z=t|z≥t). It captures prefix dependence, connects all steps to final correctness through the chain rule, localizes failure via p(zi), and creates comparable scores through consistent probability semantics. This is more structured than adding a terminal label to an otherwise local PRM.
Methodology
- �� State: st=(x,a≤t−1); the autoregressive LLM is the policy and concatenation gives deterministic transitions.
- �� Probability model: W(t) is cumulative wrong-state probability, S(t)=1−W(t), and p(t) is first-error mass.
- �� Conditional risk: h(t)=p(t)/S(t−1), the chance of failing at t given prior correctness.
- �� Outcome link: S(T)=∏t(1−h(t)) and p(t)=h(t)∏k<t(1−h(k)).
- �� Reward: with PBRS, Φ(st)=logS(t), γ=1, and intermediate reward rt=log(1−h(t)).
- �� Training: correct samples use LS=−logS(T); incorrect samples use LW=−log(1−S(T)) plus Lz=−logp(zi).
Experiments
CRM and all baselines are trained with full-parameter fine-tuning on Math-Shepherd, built from GSM8K and MATH. Comparisons include ORM, vanilla PRM, PQM, and IPRM. Best-of-N uses GSM-Plus and MATH500; beam search uses MATH500 and OOD Gaokao2023; RL uses Orz-Math-57k, Qwen2.5-Math-7B, token-level RLOO, and six zero-shot benchmarks. AUPRC tests global cross-sample comparability.
Results
CRM is consistently strong. Qwen2.5-Math-7B beam search rises from 56.07% to 64.07% on MATH500 as N grows from 4 to 100, while Gaokao2023 reaches 48.40%. In Best-of-N, Qwen2.5-3B-Instruct obtains 56.6% on MATH500 at N=32. Verifier-free RL gives CRM 43.3% on AIME24 versus PURE’s 26.6%; adding VR produces further gains with CRM+VR.
Applications
CRM can rerank sampled answers, prune prefixes in beam search, and provide dense rewards for RL without a verifier. Immediate targets include mathematical and coding assistants, where many candidate trajectories can be generated but reliable step-level checking is difficult. Deployment requires representative process data and calibrated inference budgets.
Limitations & Outlook
The method depends on Math-Shepherd-like annotations and first-error positions, which are expensive outside mathematics. Its survival-process assumption treats wrong states as irreversible, potentially mis-scoring recoverable reasoning. Full fine-tuning and per-step evaluation also incur cost. Future systems should support repair and backtracking, automate error localization, improve calibration, and test broader domains.
Plain Language Accessible to non-experts
Imagine a factory assembling a complicated product. One inspector checks only the finished item; another gives every workstation an isolated score without remembering what happened upstream. Both can fail: a later station may look productive even though an earlier mistake has already doomed the product.
CRM is a continuous quality controller. At every station it remembers the entire route so far and estimates the chance that this station is the first one to make the product unrecoverable. The chance of a successful finished product is obtained by multiplying the chances of surviving every station. Each local score therefore means something concrete: given the current history, how likely is continued success?
When a product fails, the controller also identifies the first bad station instead of blaming every station equally. That information helps choose the best route among many alternatives and teaches the workers where to improve. In the paper, this produced 64.07% on MATH500 beam search and 43.3% on AIME24 without a verifier. The idea is promising, but the factory has so far been tested mainly on mathematics and assumes mistakes cannot always be repaired.
ELI14 Explained like you're 14
Think of solving a brutal game puzzle. Every move could bring you closer to the treasure—or quietly send you into a dead end. A basic scorer checks only whether you eventually win. Another scorer grades each move separately, forgetting the route you took. That sounds unfair, right?
CRM acts like a smart teammate who remembers the whole play history. After each move it asks, “Given everything so far, how likely are we to keep winning?” A move that lowers the chance gets a lower score. The scores combine into an estimate of the whole run’s success. If you lose, CRM tries to find the first move that caused the disaster, instead of blaming every move equally.
Why is that useful? The model can generate many solution attempts and choose the most trustworthy one, or learn during training which moves to avoid. On MATH500, its beam search reached 64.07% with 100 samples. In verifier-free training it scored 43.3% on AIME24—16.7 points above PURE! The catch: it was mainly tested on math, and real reasoning can sometimes fix mistakes by backing up.
Glossary
Conditional Reward Modeling (CRM)
A reward framework that conditions each step on the complete preceding prefix and links it to final correctness. Technically, it models conditional failure probabilities rather than independent step labels.
The paper’s proposed method.
Process Reward Model (PRM)
A model that assigns feedback to intermediate reasoning steps. It provides denser supervision than an Outcome Reward Model, which scores only the final answer.
CRM’s main comparison class.
Hazard h(t)
The probability that step t is the first irreversible error, conditioned on all earlier steps being correct. Its complement is the conditional probability of maintaining correctness.
CRM’s predicted quantity.
Survival probability S(t)
The probability that reasoning has not entered a wrong state by step t. At T, it is interpreted as the probability of reaching a correct final answer.
Trajectory scoring and training losses.
Potential-Based Reward Shaping
A method that densifies sparse rewards using differences in a state potential while preserving optimal policies. CRM uses Φ(st)=logS(t).
Derivation of rt=log(1−h(t)).
Reward hacking
A failure mode in which a policy increases model-assigned reward without improving actual task success. It often exploits superficial regularities in the reward model.
CRM’s robustness motivation.
Open Questions Unanswered questions from this research
- 1 Can CRM handle recoverable mistakes, backtracking, and multiple interacting error states? Answering this requires trajectories that explicitly record correction and causal recovery.
- 2 Does probability calibration learned from Math-Shepherd transfer to code, science, and multimodal reasoning? Cross-domain process data and outcome definitions remain limited.
- 3 How can per-step inference and full fine-tuning costs be reduced? Distillation, parameter-efficient tuning, and cached prefix evaluation are plausible directions.
Applications
Immediate Applications
Mathematical answer reranking
Tutoring systems can sample multiple solutions and rank them with CRM’s S(T), rather than relying on surface-form checks or terminal ORM scores. Math-Shepherd-like process data is required; expected benefits include stronger selection on MATH500 and GSM-Plus.
Search and RL guidance
Beam search can use S(t) to retain promising prefixes, while RL can use rt as dense feedback. This is useful when exact verifiers are unavailable and may reduce incentives for verbose, templated, or superficially persuasive reasoning.
Long-term Vision
General verifier-free reasoning layer
Extended to coding, scientific analysis, and tool use, CRM could become a common process–outcome evaluation layer. Major obstacles include automatic error localization, recovery modeling, calibration, and efficient deployment at long horizons.
Abstract
Process Reward Models (PRMs) have emerged as a promising approach to enhance the reasoning capabilities of large language models (LLMs) by guiding their step-by-step reasoning toward a final answer. However, existing PRMs either treat each reasoning step in isolation, failing to capture inter-step dependencies, or struggle to align process rewards with the final outcome. Consequently, the reward signal fails to respect temporal causality in sequential reasoning and faces ambiguous credit assignment. These limitations make downstream models vulnerable to reward hacking and lead to suboptimal performance. In this work, we propose Conditional Reward Modeling (CRM) that frames LLM reasoning as a temporal process leading to a correct answer. The reward of each reasoning step is not only conditioned on the preceding steps but also explicitly linked to the final outcome of the reasoning trajectory. By enforcing conditional probability rules, our design captures the causal relationships among reasoning steps, with the link to the outcome allowing precise attribution of each intermediate step, thereby resolving credit assignment ambiguity. Further, through this consistent probabilistic modeling, the rewards produced by CRM enable more reliable cross-sample comparison. Experiments across Best-of-N sampling, beam search and reinforcement learning demonstrate that CRM consistently outperforms existing reward models, offering a principled framework for enhancing LLM reasoning. In particular, CRM is more robust to reward hacking and delivers stable downstream improvements without relying on verifiable rewards derived from ground truth.