REVES: REvision and VErification--Augmented Training for Test-Time Scaling
REVES employs a two-stage iterative framework optimizing test-time correction, improving +6.5 points on LiveCodeBench, effective for reasoning and constraint satisfaction.
Key Findings
Methodology
REVES introduces a two-phase approach: first, online data augmentation converts intermediate 'near-miss' answers into decoupled revision and verification prompts; second, single-turn RL trains the model on this augmented data, avoiding costly long-horizon sampling. The core insight is decomposing the test-time correction objective JϕSR into per-state recovery probabilities, enabling direct optimization of local correction capabilities. Empirical results show +6.5 points over RL baseline and +4.0 over standard multi-turn training on LiveCodeBench, with comparable performance to larger systems on circle packing using a 4B model. The method also enhances mathematical reasoning and out-of-distribution puzzles, demonstrating broad applicability.
Key Results
- On LiveCodeBench, REVES surpasses RL baseline by 6.5 points, and outperforms multi-turn RL by 4.0 points, showing robust improvements across test budgets.
- In circle packing, a 4B model trained with REVES matches the best results from larger evolutionary systems, indicating efficiency and effectiveness in geometric optimization.
- In math and coding tasks, REVES significantly improves correction success rates, especially under ground-truth and confidence-based stopping, confirming its correction ability and generalization.
Significance
This work addresses the fundamental misalignment between training objectives and multi-step inference dynamics in large models. By focusing on per-state correction probabilities and offline data augmentation, REVES bridges the gap, enabling models to learn effective correction strategies. Its theoretical guarantees and empirical success demonstrate a promising pathway for deploying more reliable, self-correcting AI systems in complex reasoning, planning, and constraint satisfaction tasks, with implications for both academia and industry.
Technical Contribution
The key innovation is the decomposition of the correction objective into per-state recovery probabilities, enabling direct, bias-free training. The framework combines offline data augmentation with single-turn RL, reducing computational costs and avoiding path-dependent credit assignment issues typical in multi-turn RL. Theoretical proofs guarantee transferability of improvements across various test-time algorithms that utilize revision, establishing a new paradigm for training correction-capable models.
Novelty
This is the first formalization of test-time correction as a per-state recovery probability problem, coupled with an offline data augmentation strategy that reuses intermediate states. It departs from traditional trajectory-level reward optimization, offering a more precise, efficient, and transferable training signal, thus significantly advancing the state-of-the-art in test-time model correction.
Limitations
- The approach relies on successful trajectory filtering, which may limit effectiveness when base models perform poorly or in highly novel tasks. The dependence on verification signals adds complexity, especially in real-world deployment.
- While efficient, the method still requires substantial offline computation for data augmentation, which could be challenging for extremely large models or resource-constrained settings.
- In highly complex or ambiguous tasks, the current verification mechanisms may not suffice, necessitating further research into more robust self-verification strategies.
Future Work
Future directions include integrating multi-modal feedback, extending REVES to multi-agent systems, and exploring meta-learning techniques to further improve correction adaptability. Enhancing verification robustness and reducing dependence on success filtering are also promising avenues to broaden applicability.
AI Executive Summary
REVES introduces a novel training framework designed to enhance large language models' multi-step correction capabilities during testing. Traditional training methods optimize single-shot objectives, which poorly align with the multi-turn, feedback-driven inference processes prevalent in real-world applications. To address this, REVES employs a two-stage iterative process: first, it performs offline data augmentation by running the current policy to generate successful trajectories, extracting intermediate 'near-miss' answers, and converting them into decoupled revision and verification prompts; second, it trains the model using single-turn reinforcement learning on this augmented dataset. This approach effectively decomposes the complex correction task into local, per-state recovery probabilities, enabling more precise and efficient training.
Empirical results demonstrate the effectiveness of REVES across multiple domains. On LiveCodeBench, it outperforms baseline RL by 6.5 points and standard multi-turn training by 4.0 points, showcasing robust improvements in coding tasks. In geometric optimization, a 4B model trained with REVES matches the best results from much larger evolutionary search systems, confirming its efficiency. Additionally, in mathematical reasoning and out-of-distribution puzzles, REVES significantly enhances correction success rates, confirming its broad generalization. Theoretically, the framework guarantees that improving single-step recovery probabilities transfers to overall correction performance, providing a solid foundation for future research.
This work addresses a critical gap in training for multi-step, feedback-driven inference, offering a scalable, theoretically grounded, and empirically validated solution. Its implications extend to deploying more reliable, self-correcting AI systems capable of complex reasoning, planning, and constraint satisfaction. Future research will explore integrating multi-modal signals, meta-learning, and more robust verification mechanisms, aiming to further elevate AI's autonomous correction capabilities and real-world applicability.
Deep Analysis
Background
近年来，大规模语言模型（如GPT、PaLM）在自然语言理解和生成任务中取得了巨大成功，但在复杂推理、多轮交互和约束满足任务中仍面临挑战。传统训练多依赖单次目标优化（如RLHF），难以充分捕捉多轮推理的动态特性。随着测试时修正策略（如序列修正、树搜索、演化优化）的兴起，模型在多轮交互中表现出更强的修正能力，但这些方法多依赖轨迹级奖励，存在路径偏差和采样成本高的问题。学界开始尝试将修正能力作为训练目标，结合强化学习和监督学习，提升模型的逐步修正和验证能力，但缺乏系统性框架解决多轮修正的效率和效果问题。
Core Problem
核心问题在于如何设计一种训练策略，使模型在测试时能高效、准确地进行多轮修正。现有方法多采用轨迹级奖励，容易受到路径偏差影响，且训练成本高。如何在保证训练效率的同时，显著提升模型的逐步修正能力，成为亟待解决的难题。此外，如何将中间“近失误”状态转化为有效的训练信号，也是提升模型修正能力的关键。
Innovation
本文提出REVES框架，创新点包括：1）将测试时修正目标分解为单步恢复概率，明确每个状态的修正能力；2）利用离线数据增强，将中间“近失误”答案转化为解答修正和验证提示，避免长序列采样；3）引入双阶段训练：第一阶段采样成功轨迹，生成增强数据；第二阶段用单轮RL训练模型，提升局部修正能力。这一策略有效解决路径偏差和采样成本问题，显著提升模型修正能力和泛化性。
Methodology
- �� 通过运行当前策略在训练集上采样成功轨迹，筛选出在预算内成功的轨迹，提取中间状态。
- �� 将中间状态转化为解答修正提示和验证提示，构建增强数据集。
- �� 在第二阶段，利用单轮RL在增强数据上训练模型，优化单步修正能力。
- �� 采用离线方式生成和筛选样本，避免在线长序列采样带来的高成本。
- �� 设计验证机制，允许模型在测试中自我停止，提升实际应用的鲁棒性。
Experiments
在LiveCodeBench、数学推理（MATH500、AIME24/25）和几何优化（圆包）等多个任务上，采用不同模型（如Qwen3-4B、Qwen2.5-7B）进行训练和评估。比较基线包括RL、传统多轮RL和自验证方法。指标涵盖准确率、修正成功率和任务特定的性能指标。实验还包括不同预算和停止策略的对比，验证REVES在提升修正能力和泛化能力方面的有效性。
Results
REVES在LiveCodeBench上超越RL基线6.5分，在圆包任务中用4B模型达到SOTA，表现优于大型演化搜索系统。在数学推理任务中，提升修正成功率显著，尤其在ground-truth验证和自信停止下表现优异。多任务评估显示，REVES具有良好的泛化能力，能有效应对未见任务和约束满足问题，验证了其在复杂推理场景中的潜力。
Applications
该方法适用于需要多轮交互修正的场景，如代码生成、数学推理、规划和决策支持。模型可在部署时自主判断是否修正，减少人工干预，提高自动化水平。未来可结合多模态信息和强化学习技术，拓展到更复杂的任务和实际应用中，推动大模型在工业界的落地。
Limitations & Outlook
当前方法依赖成功轨迹的筛选，可能在基础模型性能不足时效果有限。对验证机制的依赖增加了系统复杂性。训练过程中样本筛选和增强可能引入偏差，影响泛化能力。未来需优化样本效率，扩展到更复杂和多模态任务，解决在极端场景下的适应性问题。
Plain Language Accessible to non-experts
想象你在厨房里做菜，遇到一道复杂菜谱，第一次尝试可能做得不太对，但你会仔细观察哪里出了错，然后改正，再试一次。REVES就像是教厨师不断学习和改进的系统，它通过分析每次失败的原因，改进下一次的做法。它不是只看一次结果，而是不断从中间的“失误”中学习，逐步变得更擅长做出完美的菜。这种方法让厨师（模型）在面对新菜谱时，也能不断修正，最终做出令人满意的菜。
ELI14 Explained like you're 14
想象你在学校做科学实验，第一次可能做得不太对，但你会仔细观察哪里出了错，然后改正，再试一次。REVES就像是一个聪明的老师，它会帮你分析每次实验的中间步骤，找出错误的地方，告诉你怎么改。每次你修正后，它会让你再试一次，直到你做出正确的结果。这样，你就能不断学习，变得越来越擅长做科学实验。这个方法让模型也能像你一样，通过不断修正错误，变得更聪明、更厉害。
Glossary
Test-Time Scaling (测试时扩展)
在模型部署时，通过额外的修正和验证步骤提升模型性能的策略；在论文中指利用多轮修正增强模型推理能力。
REVES的核心思想是通过测试时的多轮修正实现性能提升。
Sequential Revision (序列修正)
模型在多轮交互中，根据反馈逐步修正答案的策略；是本文优化的主要目标。
本文提出的REVES框架专注于提升序列修正能力。
单步恢复概率 (One-step Recovery Probability)
在某一状态下，模型成功修正到正确答案的概率；用于分解测试目标，指导训练。
通过优化单步恢复概率，提升模型整体修正能力。
离线数据增强 (Offline Data Augmentation)
在训练过程中预先生成和筛选训练样本，避免在线长序列采样的高成本。
REVES利用离线增强提升训练效率和效果。
Verifiable Reward (可验证奖励)
在训练和测试中，利用可计算的指标（如测试用例通过情况）作为奖励信号。
论文中在数学和编码任务中采用此机制。
Open Questions Unanswered questions from this research
- 1 如何进一步提升模型在极端复杂任务中的修正能力，仍需探索更高效的样本采样和验证机制。
- 2 在多模态和多任务环境下，REVES的适应性和扩展性仍待验证。
Applications
Immediate Applications
代码自动修正
利用REVES提升AI在代码生成中的修正能力，减少错误率，增强自动调试和优化效率。
数学推理增强
在数学题解和证明中应用REVES，提升模型的推理准确率，减少错误和遗漏。
Long-term Vision
智能决策系统
未来可将REVES应用于复杂决策和规划系统，让AI自主修正偏差，提升可靠性和自主性。
Abstract
Test-time scaling via sequential revision has emerged as a powerful paradigm for enhancing Large Language Model (LLM) reasoning. However, standard post-training methods primarily optimize single-shot objectives, creating a fundamental misalignment with multi-step inference dynamics. While recent work treats this as multi-turn reinforcement learning (RL), conventional approaches optimize over the multi-step trajectories directly, failing to further exploit the high-quality mistakes in intermediate steps that model can learn from correcting them. We propose a two-stage iterative framework that alternates between online data/prompt augmentation and policy optimization. By converting the intermediate steps (``near-miss'' answers) in the successful recovery trajectories into decoupled revision and verification prompts, our approach concentrates training on both effective answer transformation and error identification. This approach enables efficient off-policy data generation and reduces the computational overhead of long-horizon sampling compared to standard multi-turn RL. On LiveCodeBench, using publicly available test cases as feedback, we observe gains of +6.5 points over the RL baseline and +4.0 points over standard multi-turn training. Beyond coding, our approach matches the previously reported SOTA result on circle packing while using the smallest base model (4B) and far fewer rollouts than the much larger evolutionary search systems. Math results under ground-truth verification further confirm improved correction ability. It also generalizes to out-of-distribution constraint-satisfaction puzzles such as n\_queens and mini\_sudoku, where correctness is defined entirely by problem constraints. Code is available at https://github.com/yxliu02/REVES.git.