EST-PRM: Stress-Testing Process Reward Models Before They Become Load-Bearing

Recent advances in reinforcement learning with human feedback (RLHF) and reward modeling have significantly improved AI reasoning capabilities. Models like OpenAI's InstructGPT and DeepMind's reward-based evaluators have demonstrated state-of-the-art performance in tasks such as dialogue, mathematical reasoning, and decision-making. These reward models typically rely on dense, step-level supervision, enabling fine-grained feedback and guiding models toward desired behaviors. However, existing evaluation benchmarks primarily assess models' correctness on natural, unperturbed data, overlooking their robustness to structural variations in reasoning chains. Prior work on reward hacking and proxy gaming (Amodei et al., 2016; Casper et al., 2023) has shown that models can exploit superficial cues, but these studies mainly focus on outcome-level rewards. Dense step-level reward models, by contrast, are more susceptible to subtle manipulations, raising concerns about their reliability and safety in real-world applications. Despite the importance, systematic stress-testing frameworks for such models are lacking, leaving a critical gap in understanding their vulnerabilities under structural perturbations.

The core challenge lies in evaluating whether dense process reward models maintain their alignment with correctness when faced with structural changes in reasoning chains that do not alter the final answer. Existing benchmarks do not test models' responses to label-preserving transformations, such as inserting redundant steps, reordering steps based on dependency constraints, or adding discourse markers. These transformations can inflate reward scores artificially or weaken the correlation between scores and true correctness, leading to potential exploitation and bias. The problem is compounded by the architecture-dependent nature of vulnerabilities, where some models are more sensitive to position changes, while others are more affected by step inflation. Addressing this requires a systematic, scalable framework capable of diagnosing these vulnerabilities and guiding improvements in reward model robustness.

This paper introduces EST-PRM, a novel stress-testing framework that applies three label-preserving transformations—step inflation, dependency-aware reordering, and confidence marker insertion—to evaluate the robustness of dense process reward models. Unlike prior benchmarks that focus solely on natural errors, EST-PRM isolates reward inflation effects and correlation degradation, providing a nuanced view of vulnerabilities. The framework employs formal metrics such as correlation collapse and reward inflation rate, grounded in theoretical analysis, to quantify how structural manipulations impact reward scoring. It enables comprehensive, architecture-specific vulnerability profiling across multiple models, revealing distinct failure modes. The approach also incorporates human validation to ensure label preservation, making it practical for real-world deployment. This systematic evaluation paradigm represents a significant advancement in reward model safety assessment.

• �� Design three label-preserving transformations: step inflation (adding redundant steps), position-sensitive reordering (permuting steps within dependency constraints), and confidence marker insertion (adding vacuous discourse cues).• Generate 4,687 reasoning chains across datasets (MATH-500, GSM8K, PRMBench) using Meta-Llama-3-70B-Instruct, ensuring diversity and correctness.• Evaluate five PRM architectures (Math-Shepherd, Qwen2.5-Math-PRM, RLHFlow, Skywork, DeepSeek) and a baseline LLM critic on these chains.• Measure reward inflation as the average score increase post-transformation, and correlation collapse via Pearson coefficient changes between scores and correctness.• Validate transformations with human annotators to confirm label preservation.• Analyze vulnerability patterns, focusing on architecture-dependent failure modes.• Propose mitigation strategies, including ensemble flagging and model fine-tuning, to enhance robustness.

The experimental setup involves generating reasoning chains from multiple datasets, injecting logical errors for control, and applying the three transformations to test model responses. Human annotators verify label preservation, ensuring that transformations do not alter correctness labels. Metrics include Pearson correlation changes to quantify sensitivity, reward inflation rates to detect score gaming, and visualizations like score distribution plots. The experiments compare model vulnerabilities across different architectures, revealing that some models are more sensitive to position shifts, while others are prone to reward inflation during step inflation. Additional controls, such as random filler insertion, differentiate structural vulnerabilities from random noise. The robustness of mitigation strategies is evaluated on a held-out calibration set, balancing false positives and detection sensitivity.

The results demonstrate architecture-specific vulnerabilities: Math-Shepherd is most sensitive to position perturbations, with a correlation drop of 0.152±0.038 and a reward inflation of 32.8±4.9%. Qwen2.5-Math-PRM shows the highest reward inflation during step inflation, reaching 47.6±4.3%. Models like RLHFlow exhibit minimal correlation collapse, indicating different structural biases. Visual analyses reveal that attacks blur the separation between correct and corrupted chains, confirming the theoretical predictions. Mitigation strategies improve robustness but involve trade-offs, such as increased false positives. Overall, the findings highlight that reward models are vulnerable to structural manipulations, necessitating more comprehensive evaluation frameworks.

The framework can be employed to evaluate and improve reward models used in high-stakes AI systems, including automated reasoning, medical diagnosis, and financial decision-making. By identifying structural vulnerabilities, developers can refine models to prevent reward gaming and bias exploitation. The approach also supports designing more robust reinforcement learning pipelines, ensuring that reward signals remain aligned with true correctness even under adversarial manipulations. Long-term, this work can inform standards for AI safety and robustness, fostering trustworthy AI deployment across diverse domains.

The current evaluation focuses mainly on mathematical reasoning tasks, limiting generalization to other domains like commonsense or dialogue. The transformations, while diverse, do not cover all possible structural attacks, leaving some vulnerabilities untested. Computational costs for large-scale stress testing are high, restricting real-time application. The mitigation strategies, though effective, may increase false positives or reduce sensitivity, requiring further refinement. Future work should explore broader tasks, optimize algorithms for efficiency, and develop adaptive defenses to ensure comprehensive robustness.