Rethinking Reward Models for Multi-Domain Test-Time Scaling
This paper compares four reward models (dORM, dPRM, gORM, gPRM) across 14 multi-domain tasks, finding gORM most robust and effective.
Key Findings
Methodology
Authors evaluate four reward model variants—discriminative and generative ORM and PRM—using a unified protocol across diverse datasets like PRM800K and MMLU-Pro. They train models on multi-domain data, then assess performance via F1, accuracy, and N metrics, analyzing error accumulation in long reasoning chains and robustness to label noise. The study combines theoretical analysis with empirical validation, including ablation on chain length and noise sensitivity, to understand model behavior in complex scenarios.
Key Results
- In math tasks, dPRM outperforms dORM, and gORM surpasses gPRM, with gORM showing a 5-10% F1 score improvement and better stability in long chains. In multi-domain settings, gORM maintains high performance, with an 8-12% advantage over discriminative models, especially under noisy labels. Longer reasoning chains lead to error accumulation in PRMs, while gORM demonstrates robustness against label noise, confirming its suitability for real-world deployment.
- Across 14 domains, gORM consistently outperforms others, with notable gains in specialized fields like law and medicine. The results highlight that stepwise aggregation in PRMs amplifies errors as reasoning length increases, whereas generative models mitigate this effect, offering more reliable verification. The experiments also show that multi-domain training enhances model generalization, with gORM requiring fewer samples to achieve high accuracy.
- The findings challenge the conventional wisdom that finer supervision (PRM) always yields better results. Instead, the study advocates for outcome verification via generative models, especially in complex, multi-domain environments, where label noise and long reasoning trajectories pose significant challenges.
Significance
This work shifts the paradigm in reward model design, emphasizing robustness and scalability over fine-grained supervision. It provides practical insights for deploying LLMs in high-stakes domains like healthcare and law, where reliability is critical. By demonstrating that generative outcome verification (gORM) outperforms traditional discriminative approaches in diverse settings, it opens pathways for more resilient AI systems. The research also offers theoretical foundations explaining error propagation, guiding future improvements in reward modeling and test-time scaling strategies.
Technical Contribution
The paper introduces a comprehensive comparison of four reward model variants, integrating theoretical analysis of error propagation with empirical validation across multiple datasets. It reveals that stepwise scoring in PRMs inherits label noise and amplifies errors with chain length, while generative models like gORM are inherently more noise-robust. The study develops new training protocols for multi-domain scenarios, demonstrating that outcome-based, generative reward models are more scalable and reliable, especially when automatic labeling introduces noise. These contributions advance the understanding of reward model dynamics and practical deployment in complex environments.
Novelty
This is the first systematic evaluation of discriminative versus generative reward models across a broad set of multi-domain tasks, extending beyond math-centric benchmarks. It uncovers the counterintuitive finding that generative models (gORM) outperform discriminative ones in real-world scenarios, challenging the assumption that fine-grained supervision (PRM) is always superior. The integration of theoretical error analysis with extensive empirical testing provides a new perspective on reward model robustness, especially under label noise and long reasoning chains.
Limitations
- Despite robustness, gORM and gPRM still face challenges with extremely long chains and self-correcting reasoning, where error propagation can occur. The reliance on automatic labels, which may contain biases, limits performance in certain domains. Additionally, training these models requires substantial computational resources, and their effectiveness depends on the quality of generated verification CoTs. Future work should focus on improving label quality, reducing computational costs, and extending evaluations to more complex, real-world scenarios.
Future Work
Future directions include integrating human feedback to refine reward models, developing multi-modal verification strategies, and exploring reinforcement learning techniques to further enhance robustness. Additionally, expanding evaluations to more complex, high-stakes domains like legal and medical fields, and designing adaptive models that can handle dynamic data distributions, will be crucial. The goal is to create scalable, trustworthy AI systems capable of reliable reasoning in diverse real-world applications.
AI Executive Summary
As large language models (LLMs) become increasingly integral to complex reasoning tasks, ensuring their reliability during test-time scaling remains a critical challenge. Traditional reward models, especially outcome reward models (ORM), have primarily focused on evaluating the final answer, offering coarse supervision. However, recent advancements introduced process reward models (PRM), which score intermediate reasoning steps, promising finer-grained feedback. Despite this, the effectiveness of PRMs in multi-domain, real-world scenarios has not been thoroughly validated.
This study conducts a comprehensive comparison of four reward model variants—discriminative ORM (dORM), discriminative PRM (dPRM), generative ORM (gORM), and generative PRM (gPRM)—across 14 diverse domains, including mathematics, law, and medicine. Using a unified experimental protocol, the authors evaluate these models on datasets like PRM800K and MMLU-Pro, analyzing their performance in terms of accuracy, F1 score, and robustness to label noise and long reasoning chains. The results reveal that, contrary to prior assumptions, generative models, particularly gORM, consistently outperform discriminative models in multi-domain settings.
A key insight from the research is that stepwise scoring in PRMs inherits label noise from automatic annotations, leading to error accumulation as reasoning chains lengthen. In contrast, gORM demonstrates remarkable robustness, maintaining high performance even under noisy labels and extended reasoning trajectories. These findings challenge the conventional belief that fine-grained supervision is always better, advocating instead for outcome verification via generative models for scalable, multi-domain deployment.
The implications are significant: deploying gORM in high-stakes environments like healthcare and legal systems can enhance trustworthiness and reliability. The study also provides theoretical analysis explaining error propagation mechanisms, guiding future improvements. Limitations include challenges with extremely long chains and reliance on automatic labels, which may contain biases. Future work will focus on integrating human feedback, reducing computational costs, and extending evaluations to more complex, real-world applications, aiming to develop scalable, trustworthy AI reasoning systems.
Deep Dive
Abstract
The reliability of large language models (LLMs) during test-time scaling is often assessed with \emph{external verifiers} or \emph{reward models} that distinguish correct reasoning from flawed logic. Prior work has studied both outcome reward models (ORMs), which assess only the final answer, and process reward models (PRMs), which score intermediate reasoning steps. Although PRMs are often viewed as advantageous due to their finer-grained supervision, much of the supporting evidence comes from math-adjacent settings, and their relative benefits across broader domains remain unclear. We present the first unified evaluation of four reward model variants, discriminative ORM and PRM (dORM, dPRM) and generative ORM and PRM (gORM, gPRM), across 14 diverse domains. Contrary to conventional wisdom, we find that (i) dORM performs on par with dPRM, (ii) gPRM is not competitive, and (iii) overall, gORM is the most robust, yielding significant and consistent gains across every tested domain. We attribute the worse performance of gPRM to the stepwise scoring process, which inherits label noise from LLM-based automatic labeling, leading to difficulties in evaluating long reasoning trajectories, including those involving self-correcting reasoning. Both our theoretical analysis and empirical observations indicate that stepwise aggregation compounds errors as reasoning length increases. These findings challenge the common assumption that fine-grained supervision is always better and support generative outcome verification for multi-domain deployment. Our \href{https://github.com/db-Lee/Multi-RM}{\underline{code}} is publicly available to facilitate future research in multi-domain settings.