Score Centering Stabilizes Off-policy Reinforcement Learning
Score Centering stabilizes off-policy RL by reducing training-inference mismatch drift.
Key Findings
Methodology
The paper introduces Score Centering, a method that stabilizes RL by canceling drift between training and inference engines. It achieves this by subtracting the sampler's expected score at each prefix.
Key Results
- Experiments on the Qwen3-30B-A3B-Base model show Score Centering outperforms importance sampling under heavy quantization, achieving 58% training accuracy.
- On the INTELLECT-2 dataset, Score Centering combined with importance sampling outperforms pure importance sampling baselines in staleness experiments.
- In synthetic weight noise experiments, Score Centering remains stable under the most severe noise conditions.
Significance
This research effectively addresses the instability in RL caused by training-inference mismatch, particularly in large-scale language model training. It offers academia and industry a new method for stable training, reducing computational resource waste.
Technical Contribution
Score Centering provides a novel bias correction method, differing from existing importance sampling methods by being a deterministic additive correction term, reducing gradient variance.
Novelty
Score Centering is the first bias correction method that does not rely on importance ratios, offering a new mechanism for bias correction compared to traditional methods.
Limitations
- Score Centering may not perform as well under extreme staleness conditions compared to its combination with importance sampling.
- In certain quantization settings, Score Centering may need to be combined with other correction methods for optimal performance.
Future Work
Future research directions include exploring the combination effects of Score Centering with other correction methods and its applicability across different models and datasets.
AI Executive Summary
Reinforcement learning is crucial for training large language models, but small differences between training and inference engines often lead to instability. Existing methods struggle to completely eliminate this mismatch, while Score Centering stabilizes training by canceling drift. Experiments demonstrate its superiority over importance sampling under quantization and staleness conditions, especially in large-scale models. Score Centering offers academia and industry a new method for stable training, reducing computational resource waste. However, it may not perform as well under extreme staleness conditions compared to its combination with importance sampling. Future research will explore its applicability across different models and datasets.
Deep Analysis
Background
With the rise of large language models, reinforcement learning has become an essential step in training. However, small numerical differences between training and inference engines, known as training-inference mismatch, often lead to instability. Existing methods like importance sampling can correct bias but increase gradient variance.
Core Problem
Training-inference mismatch arises from numerical differences between training and inference engines, especially severe under quantization and staleness conditions. This mismatch can lead to unstable training or even reward collapse.
Innovation
Score Centering cancels drift by subtracting the sampler's expected score at each prefix, stabilizing training. This method does not rely on importance ratios, reducing gradient variance.
Methodology
- �� Compute sampler's expected score
- �� Subtract expected score at each prefix for Score Centering
- �� Combine Score Centering with importance sampling for enhanced stability
Experiments
Experiments conducted on the Qwen3-30B-A3B-Base model tested Score Centering's performance under quantization and staleness conditions. The INTELLECT-2 dataset was used for training, comparing the effects of different correction methods.
Results
Score Centering outperforms importance sampling under heavy quantization, achieving 58% training accuracy. In staleness experiments, Score Centering combined with importance sampling outperforms pure importance sampling baselines.
Applications
Score Centering can be used for stable training of large-scale language models, reducing computational resource waste and improving training efficiency.
Limitations & Outlook
Score Centering may not perform as well under extreme staleness conditions compared to its combination with importance sampling. Future research will explore its applicability across different models and datasets.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen, and Score Centering is like ensuring every dish tastes consistent. The differences between training and inference engines are like different chefs' techniques, and Score Centering adjusts the seasoning to ensure consistent flavor.
ELI14 Explained like you're 14
Imagine you're playing a game, and Score Centering is like making sure every attack hits the target accurately. The differences between training and inference engines are like different weapons' accuracy, and Score Centering adjusts the aim to ensure hit rate.
Glossary
Training-Inference Mismatch
Refers to numerical differences between training and inference engines causing bias.
In this paper, this mismatch leads to instability in reinforcement learning.
Score Centering
A method that cancels drift by subtracting the sampler's expected score.
Used to stabilize reinforcement learning training.
Importance Sampling
A method that corrects bias by multiplying by importance ratios.
In this paper, combined with Score Centering for enhanced stability.
Quantization
Refers to reducing model parameters and computation precision to save resources.
In this paper, quantization is a source of training-inference mismatch.
Staleness
Refers to bias caused by delayed updates of inference engines.
In this paper, Score Centering combined with importance sampling performs better under extreme staleness conditions.
Open Questions Unanswered questions from this research
- 1 How can Score Centering be optimized under extreme staleness conditions?
- 2 How to apply Score Centering across different models and datasets?
Applications
Immediate Applications
Large-scale Language Model Training
Score Centering can be used for stable training of large-scale language models, reducing computational resource waste.
Long-term Vision
Intelligent System Optimization
Score Centering can optimize the training process of intelligent systems, improving efficiency and stability.
Abstract
Reinforcement learning (RL) of large language models is notoriously sensitive to small differences between training and inference engines, often referred to as the training-inference mismatch (TIM). However, completely eliminating TIM is impractical, as it would come at a major cost to rollout efficiency. In this paper, we show that the instability of RL under TIM is primarily caused by drift: a persistent bias between training and inference engines that accumulates with every training step. We derive an additive "score centering" correction term that stabilizes RL under TIM by canceling drift. When training models from 0.6B to 30B parameters, score centering alone matches or outperforms methods based on importance sampling under quantization, with the gap growing as the mismatch becomes more severe. Because the correction is additive, score centering also composes with importance sampling -- their composition outperforms pure importance-sampling baselines in our staleness experiments.