When to Vote, When to Rewrite: Disagreement-Guided Strategy Routing for Test-Time Scaling
Proposed a disagreement-guided strategy routing method, improving mathematical reasoning accuracy by 3%-7%.
Key Findings
Methodology
The method is training-free, utilizing output disagreement as a signal to dynamically choose inference strategies. The framework includes lightweight resolution, majority voting, and rewriting-based reformulation for different levels of disagreement.
Key Results
- Improved accuracy by 3%-7% on seven mathematical benchmarks, especially effective on challenging benchmarks.
- Reduced sampling cost compared to existing methods, notably on Math500 and Olympiad benchmarks.
- Performed well on code generation tasks, demonstrating the method's generality.
Significance
This research provides an efficient test-time scaling method for mathematical reasoning tasks, significantly improving model performance on challenging instances and reducing unnecessary computational resource waste.
Technical Contribution
Introduced a novel inference strategy selection framework that uses output disagreement as an uncertainty signal, avoiding computational waste of traditional single-strategy methods.
Novelty
First to use output disagreement to guide inference strategy selection, differing from traditional methods that allocate computation within a single strategy.
Limitations
- Rewriting may degrade performance on simple instances as it can alter the semantic meaning of the problem.
- The method relies on the accuracy of disagreement detection, which may fail in some cases.
Future Work
Future work could explore applying this strategy routing framework to more tasks and optimizing the accuracy of disagreement detection.
AI Executive Summary
Large reasoning models excel in mathematical reasoning tasks but remain unreliable on complex instances. Existing test-time scaling methods like repeated sampling, self-correction, and tree search, while effective, often show diminishing returns on hard problems.
This paper proposes a novel training-free framework that formulates test-time scaling as an instance-level routing problem rather than allocating more computation within a single strategy. The framework dynamically selects among different scaling strategies based on output disagreement: lightweight resolution for consistent cases, majority voting for moderate disagreement, and rewriting-based reformulation for highly ambiguous instances.
Experimental results show that this method improves accuracy by 3% to 7% on seven mathematical benchmarks and three models while reducing sampling costs. This research provides an efficient test-time scaling method for mathematical reasoning tasks, significantly improving model performance on challenging instances and reducing unnecessary computational resource waste.
Deep Analysis
Background
Large reasoning models perform well on mathematical reasoning tasks but struggle with complex instances. Existing test-time scaling methods like repeated sampling, self-correction, and tree search, while effective, often show diminishing returns on hard problems.
Core Problem
How to dynamically select appropriate inference strategies at test time to improve model performance on complex instances and reduce unnecessary computational resource waste.
Innovation
Proposed a strategy routing framework based on output disagreement, dynamically selecting different inference strategies, avoiding computational waste of traditional single-strategy methods.
Methodology
- �� Use output disagreement as an uncertainty signal
- �� Lightweight resolution for consistent cases
- �� Majority voting for moderate disagreement
- �� Rewriting-based reformulation for highly ambiguous instances
Experiments
Conducted experiments on seven mathematical benchmarks and three models, comparing the performance of various test-time scaling methods, validating the method's effectiveness and generality.
Results
Improved accuracy by 3%-7% on seven mathematical benchmarks, especially effective on challenging benchmarks. Reduced sampling cost compared to existing methods.
Applications
Applicable to mathematical reasoning tasks, especially scenarios requiring dynamic inference strategy selection.
Limitations & Outlook
The method relies on the accuracy of disagreement detection, which may fail in some cases.
Plain Language Accessible to non-experts
Imagine a factory with different production lines for different products. This method is like an intelligent system that can dynamically choose the most suitable production line based on the complexity of the product, improving efficiency and reducing waste.
ELI14 Explained like you're 14
Imagine you're playing a puzzle game. Sometimes you encounter really hard puzzles. This method is like a smart assistant that can choose different strategies based on the puzzle's difficulty to help you solve it. Isn't that cool?
Glossary
Large Reasoning Models
Models with strong reasoning capabilities, typically used for complex tasks.
Perform well on mathematical reasoning tasks but struggle with complex instances.
Test-Time Scaling
Methods that allocate additional computation at inference time to improve output quality.
Existing methods include repeated sampling, self-correction, and tree search.
Disagreement
Inconsistency among model outputs, often used to assess model uncertainty.
Used as a signal for selecting inference strategies.
Rewriting
Method of improving model performance by altering problem formulation.
Used for highly ambiguous instances.
Majority Voting
Method of selecting the final answer by voting among multiple outputs.
Used for moderate disagreement cases.
Open Questions Unanswered questions from this research
- 1 How to improve the accuracy of disagreement detection to better select inference strategies.
- 2 Feasibility and effectiveness of applying this framework to other tasks.
Applications
Immediate Applications
Mathematical Reasoning Tasks
Can dynamically select inference strategies in mathematical reasoning tasks, improving model performance on complex instances.
Long-term Vision
General Inference Framework
Apply this framework to more tasks, exploring its potential in various fields.
Abstract
Large Reasoning Models (LRMs) achieve strong performance on mathematical reasoning tasks but remain unreliable on challenging instances. Existing test-time scaling methods, such as repeated sampling, self-correction, and tree search, improve performance at the cost of increased computation, yet often exhibit diminishing returns on hard problems. We observe that output disagreement is strongly correlated with instance difficulty and prediction correctness, providing a useful signal for guiding instance-level strategy selection at test time. Based on this insight, we propose a training-free framework that formulates test-time scaling as an instance-level routing problem, rather than allocating more computation within a single strategy, dynamically selecting among different scaling strategies based on output disagreement. The framework applies lightweight resolution for consistent cases, majority voting for moderate disagreement, and rewriting-based reformulation for highly ambiguous instances. Experiments on seven mathematical benchmarks and three models show that our method improves accuracy by 3% - 7% while reducing sampling cost compared to existing approaches.