Adversarial Training for Process Reward Models

In recent years, the field of machine learning has made significant progress in complex reasoning capabilities, particularly in large language models (LLMs). These models have performed exceptionally well on complex problems such as the International Mathematical Olympiad, yet their reasoning capabilities remain fundamentally unreliable, often producing factually incorrect outputs despite appearing coherent. This unreliability is critical in high-stakes domains such as scientific discovery, finance, or medicine, where subtle errors can have severe consequences. Process Reward Models (PRMs) have been proposed to identify very subtle errors to enhance reasoning trustworthiness.

Current PRM training techniques predominantly rely on static datasets, which provide a fixed error distribution that cannot adapt to more nuanced errors. Additionally, synthetic data generation techniques assume that correct final answers imply correct intermediate steps, lacking mechanisms to actively mine harder negatives. To address these issues, a PRM training paradigm that provides an adaptive curriculum is required, where negative sample hardness dynamically increases while aligning with the PRM's evolving capabilities.

This paper introduces Adversarially Trained Process Reward Models (APRM), a novel adversarial training method. APRM models PRM training as a two-player non-cooperative game, where a generator learns to produce reasoning errors to deceive a reward model, while the reward model concurrently learns to detect these errors. This competitive interaction creates the desired adaptive curriculum, directly addressing the limitations of static or passively generated training data.

• �� APRM models reward model training as a two-player non-cooperative game, where a generator learns to produce reasoning errors to deceive a reward model.
• �� The reward model concurrently learns to detect these errors, yielding progressively harder negatives.
• �� Game-aware optimizers and symmetric policy regularization ensure stable training dynamics with provable linear-rate convergence to Nash Equilibrium.
• �� Empirical validation shows APRM outperforms state-of-the-art PRM training and prompting methods on mathematical reasoning benchmarks.

The experimental design includes training on the MATH dataset, where the generator learns to perturb solution steps. We evaluate on five mathematical reasoning benchmarks: MATH500, JEEBench, OlympiadBench, AIME25, and AMC. We compare APRM against existing methods with comparable test-time compute, including prompting methods and trained reward models. Training is conducted using PPO with Optimistic Gradient Descent-Ascent (OGDA).

APRM improves solver accuracy by 3.4 percentage points on various mathematical reasoning benchmarks compared to the strongest PRM baseline. Notably, APRM achieves a 5.3 percentage point improvement on out-of-distribution tasks, demonstrating superior generalization capabilities. Ablation studies reveal that removing entropy regularization and OGDA optimizer significantly reduces model performance, confirming the importance of these components for stable training.

APRM can be directly applied to mathematical reasoning tasks, enhancing solver accuracy and robustness. Its superior performance on out-of-distribution tasks indicates strong adaptability to novel errors. This is particularly important for trustworthy reasoning in high-stakes domains such as scientific discovery, finance, and medicine.

APRM may perform poorly in non-mathematical domains as its training data primarily comes from mathematical reasoning benchmarks. Additionally, the model may struggle to identify errors in complex problems involving multiple steps. Future research directions include extending APRM to more domains such as scientific and engineering reasoning and enhancing the generator's ability to produce more challenging negatives.