Single-Rollout Hidden-State Dynamics for Training-Free RLVR Data Selection
SHIFT method selects data using inference-time hidden-state dynamics, enhancing RLVR performance.
Key Findings
Methodology
SHIFT selects data using inference-time hidden-state dynamics without training or labels. It computes the reasoning-induced representation shift (RIRS) as a proxy for instance utility and ensures coverage via a quality-weighted farthest-first CoreSet strategy.
Key Results
- In mathematical reasoning and medical QA benchmarks, SHIFT performs excellently under ultra-low budgets, enhancing in-domain accuracy and transfer to harder evaluation settings. For example, SHIFT achieved 62.67 accuracy on MATH-500 with 2% data selection.
- SHIFT outperformed the full-data RLVR reference on AMC, indicating that selecting high-impact examples can improve OOD generalization.
- Ablation studies show that RIRS-based coverage and quality-weighting provide complementary gains, and analyses indicate that RIRS is not explained by simple input/output length statistics.
Significance
SHIFT addresses the bottleneck of data selection in RLVR without labels or training signals. It significantly reduces reward/annotation and training overhead under severe supervision scarcity, enhancing both in-domain and cross-domain performance.
Technical Contribution
SHIFT offers new theoretical guarantees and engineering possibilities by selecting data using inference-time hidden-state dynamics, unlike existing methods that rely on training-time signals or labels, significantly reducing computational costs.
Novelty
SHIFT is the first to propose data selection using inference-time hidden-state dynamics, offering a new perspective and solution compared to existing methods that rely on training signals.
Limitations
- SHIFT may not capture all influencing factors in complex domains, especially those requiring more intricate inference dynamics.
- The method's performance under ultra-low budgets needs further validation, particularly on datasets from different domains.
Future Work
Future work could explore the application of SHIFT in more domains and study how to integrate other unsupervised signals to further enhance selection effectiveness.
AI Executive Summary
Reinforcement learning with verifiable rewards (RLVR) is a powerful paradigm for enhancing the reasoning ability of large language models, but its sensitivity to data selection makes it a bottleneck. SHIFT selects data using inference-time hidden-state dynamics without training or labels. It computes the reasoning-induced representation shift (RIRS) as a proxy for instance utility and ensures coverage via a quality-weighted farthest-first CoreSet strategy. Experiments show that SHIFT performs excellently in mathematical reasoning and medical QA benchmarks, especially under ultra-low budgets. SHIFT not only enhances in-domain accuracy but also improves cross-domain generalization. However, the method may not capture all influencing factors in complex domains, and future work could explore more domain applications. SHIFT offers a new perspective for data selection, significantly reducing reward/annotation and training overhead under severe supervision scarcity.
Deep Analysis
Background
Reinforcement learning with verifiable rewards (RLVR) has recently emerged as a powerful paradigm for enhancing the reasoning ability of large language models. Existing studies suggest that RLVR can be extremely data-efficient, but its sensitivity to chosen instances makes data selection a key bottleneck. Most existing selection pipelines rely on training-time optimization signals or require access to verifiable rewards or ground-truth answers over large candidate pools, which is costly and often infeasible in specialized domains.
Core Problem
The core problem is how to efficiently identify impactful RL training examples without labels or reward evaluation. Existing methods typically rely on training-time signals or labels, limiting their applicability in truly low-resource settings.
Innovation
SHIFT selects data using inference-time hidden-state dynamics without training or labels. It computes the reasoning-induced representation shift (RIRS) as a proxy for instance utility and ensures coverage via a quality-weighted farthest-first CoreSet strategy.
Methodology
- �� SHIFT selects data using inference-time hidden-state dynamics without training or labels.
- �� Computes the reasoning-induced representation shift (RIRS) as a proxy for instance utility.
- �� Ensures coverage via a quality-weighted farthest-first CoreSet strategy.
- �� Produces compact subsets that scale to large unlabeled pools.
Experiments
Experiments are conducted on mathematical reasoning and medical QA benchmarks using ultra-low budgets for data selection. SHIFT's performance is evaluated in terms of in-domain accuracy and cross-domain generalization. Baselines include random selection, diversity, and difficulty/uncertainty heuristics.
Results
SHIFT performs excellently in mathematical reasoning and medical QA benchmarks, especially under ultra-low budgets. SHIFT not only enhances in-domain accuracy but also improves cross-domain generalization. Ablation studies show that RIRS-based coverage and quality-weighting provide complementary gains.
Applications
SHIFT can be applied in domains requiring efficient data selection, such as medical QA and mathematical reasoning. Its training-free and label-free characteristics make it advantageous in low-resource environments.
Limitations & Outlook
SHIFT may not capture all influencing factors in complex domains, especially those requiring more intricate inference dynamics. The method's performance under ultra-low budgets needs further validation, particularly on datasets from different domains.
Plain Language Accessible to non-experts
Imagine you're in a kitchen trying to cook a meal. You have many ingredients but don't know which ones will make the best dish. SHIFT is like a smart chef who can pick the most promising ingredients without trying each one. It observes the changes in the ingredients to decide which ones will make the best meal. This approach saves time and ensures a delicious dish.
ELI14 Explained like you're 14
Imagine you're playing a game where you need to choose characters to form a team. SHIFT is like a super helper that tells you which characters have the most potential without playing each one. It observes the characters' performance to decide which ones will help you win the game. This approach saves time and boosts your chances of winning! Isn't that cool?
Glossary
Reinforcement Learning
A machine learning method that trains models through rewards and punishments.
Used in this paper to enhance language model reasoning abilities.
Representation Shift
Changes in hidden states during inference, used to evaluate instance utility.
Key metric for data selection in the SHIFT method.
CoreSet
A selection algorithm ensuring good coverage of the chosen subset.
Used in SHIFT to ensure diversity in data selection.
Inference Dynamics
Changes in model hidden states during inference.
Used in SHIFT to evaluate data selection utility.
Medical QA
A question-answering task involving medical domain questions.
One of the benchmarks used to evaluate the SHIFT method.
Open Questions Unanswered questions from this research
- 1 How to apply SHIFT in more complex inference dynamics remains to be explored.
- 2 SHIFT's performance on datasets from different domains needs further validation.
Applications
Immediate Applications
Medical QA
SHIFT can be used in medical QA tasks to enhance model performance through efficient data selection.
Long-term Vision
Cross-domain Data Selection
SHIFT can be applied to more domains for data selection, enhancing model generalization capabilities.
Abstract
Reinforcement learning with verifiable rewards (RLVR) can yield large reasoning gains from very few training instances, yet its strong sensitivity to which instances are used makes data selection a central bottleneck. Most existing selection pipelines rely on training-time optimization signals and/or require access to verifiable rewards or ground-truth answers over large candidate pools, which is costly and often infeasible in specialized domains. We study RLVR data selection in a setting where selection must be performed before any RL training and without labels or reward evaluation on the full pool. We propose SHIFT, a one-shot, training-free selector based solely on inference-time hidden-state dynamics. For each candidate instance, SHIFT runs a single deterministic reasoning rollout and computes a reasoning-induced representation shift (RIRS) as the start-to-end hidden-state delta. SHIFT uses the RIRS magnitude as a lightweight proxy for instance utility and enforces coverage via a quality-weighted farthest-first CoreSet procedure in an RIRS-augmented feature space, producing compact subsets that scale to large unlabeled pools. Across mathematical reasoning and medical QA benchmarks under ultra-low budgets, SHIFT consistently outperforms training-free diversity and difficulty/uncertainty baselines, improving both in-domain accuracy and transfer to harder evaluation settings. Ablations show that RIRS-based coverage and quality-weighting contribute complementary gains, and analyses indicate that RIRS is not explained by simple input/output length statistics. Code is available at github.com/JianghaoWu/SHIFT.