SCAN: Self-Denoising Monte Carlo Annotation for Robust Process Reward Learning

The evolution of large language models (LLMs) such as GPT-4 and PaLM has driven interest in enhancing their reasoning capabilities. Process reward models (PRMs) evaluate models step-by-step, improving interpretability and self-correction. Early efforts like Chain-of-Thought prompting demonstrated the importance of intermediate reasoning steps. However, high-quality data for training PRMs remains scarce due to expensive human annotation. Synthetic data via Monte Carlo (MC) estimation emerged as a cost-effective alternative, but its high noise level hampers performance. Recent approaches incorporated external critic models (e.g., Qwen-72B) for noise filtering, yet these depend on additional supervision. This paper addresses the core challenge: how to effectively denoise MC-generated data without external supervision, enabling scalable, robust PRM training.

The key issue is that MC estimation produces noisy labels due to the imperfect nature of the completer models, which tend to both underestimate and overestimate step correctness. This bias leads to overfitting, poor generalization, and limited scalability of PRMs trained solely on synthetic data. Existing solutions rely on external critic models or large human datasets, which are costly and less flexible. The challenge is to develop a method that can automatically identify and correct noise within synthetic data, maintaining high annotation quality while minimizing computational costs. Addressing this problem is crucial for deploying PRMs in real-world applications requiring large, diverse datasets.

The paper introduces a comprehensive framework combining noise distribution analysis, self-confidence metrics, and self-correcting training strategies. Key innovations include: 1) analyzing the bias patterns in MC-generated data, revealing under- and over-estimation issues; 2) designing a self-confidence score to evaluate annotation reliability; 3) filtering samples based on confidence to focus training on high-quality data; 4) employing a self-denoising loss that adaptively corrects biased labels; 5) leveraging lightweight models for efficient data synthesis, drastically reducing inference costs. These contributions enable high-quality, scalable PRM training without external critic models, setting a new standard for noise-robust learning.

• �� Analyze noise distribution in MC synthetic data, identifying bias types (under- and over-estimation). • Develop a self-confidence metric SCθ(q) to measure annotation reliability. • Use this metric to filter high-confidence samples for positive data, reducing noise. • Collect negative samples with uncertain annotations for step-wise correctness annotation via a lightweight generator model. • Apply Monte Carlo estimation selectively to these uncertain samples, maximizing data utilization. • Introduce a self-correcting loss function that reweights labels based on confidence scores, mitigating bias. • Use lightweight models like Qwen2.5-Math-1.5B for response generation, lowering inference costs. • Tune confidence thresholds and tolerance distance d for optimal noise filtering. • Train the PRM with reweighted labels, improving robustness against noisy annotations.

The experimental setup involves training PRMs on synthetic datasets generated by lightweight models, primarily Qwen2.5-Math-1.5B and 7B, using ProcessBench, GSM8K, and Numina-Math as benchmarks. The core evaluation metrics include F1 scores, error detection accuracy, and best-of-N response selection. The experiments compare traditional MC estimation, external critic filtering, and the proposed SCAN framework, with ablation studies on confidence thresholds, sample size, and loss functions. Hyperparameters such as response count (k=64 or 128) and tolerance distance (d=2) are optimized. Results show that models trained with SCAN outperform baselines trained on large human datasets, with significant improvements in both accuracy and robustness across multiple reasoning benchmarks.

The PRM trained with 101K synthetic samples using SCAN achieves an F1 of 59.1 on ProcessBench, a substantial increase from 19.9 baseline, surpassing models trained on 800K human-annotated data. Performance scales positively with data volume, reaching over 197K samples, with continued improvements. The approach effectively filters noisy samples via self-confidence, leading to more reliable training data. Ablation studies confirm the importance of confidence thresholds and the self-denoising loss. The lightweight models demonstrate that high-quality annotations are feasible at low inference costs, making large-scale training practical. Overall, the results validate the framework’s scalability, robustness, and efficiency.

SCAN enables cost-effective training of PRMs for complex reasoning tasks such as automated mathematics, legal reasoning, and scientific analysis. Its low-cost synthetic data generation makes it accessible for industry and academia, reducing dependence on expensive human annotations. The framework can be integrated into multi-task learning pipelines, supporting scalable AI systems in education, research, and decision-making. Long-term, it paves the way for autonomous AI agents capable of deep reasoning with minimal supervision, facilitating deployment in real-world scenarios like intelligent tutoring, legal document analysis, and scientific discovery.

Current reliance on self-confidence metrics may limit effectiveness in highly noisy or ambiguous scenarios. The method's sensitivity to hyperparameters like confidence thresholds and tolerance distance requires careful tuning. Scalability to multi-modal data or tasks involving vision, audio, or sensor inputs remains to be validated. The computational cost, although reduced, still involves multiple response sampling and MC estimation, which could be challenging for extremely large datasets. Future work should focus on adaptive parameter tuning, multi-modal extension, and theoretical analysis of noise correction guarantees.