RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation
RLCSD combines contrastive learning with reinforcement on-policy self-distillation, effectively mitigating style drift and improving reasoning model performance.
Key Findings
Methodology
This paper introduces RLCSD (Reinforcement Learning with Contrastive on-policy Self-Distillation), a novel framework that integrates contrastive learning into reinforcement learning-based self-distillation to address style drift issues. The core concept involves sampling multiple rollouts for each query, using a verifier to classify them into correct and incorrect groups, and then constructing a contrastive token-level signal by comparing the teacher's distribution conditioned on correct versus incorrect hints. This contrastive difference (ectr) suppresses shared stylistic components, emphasizing task-relevant tokens. The contrastive signal is then normalized and modulated to serve as a per-token adjustment factor, which modulates the advantage estimate (AORM) in a two-path PPO setup, ensuring stable training. Extensive experiments on large models like Qwen3 (1.7B, 4B, 8B) and Olmo-3-7B-Think across mathematical and logical reasoning tasks demonstrate that RLCSD consistently outperforms baseline methods such as GRPO, SDPO, and SRPO, with improved stability and performance.
Key Results
- On the Qwen3 series, RLCSD achieves a 3-4% increase in pass@1 accuracy on math reasoning tasks, reaching 88.8%, 74.2%, and 66.9% respectively, outperforming GRPO and prior OPSD methods. In logical reasoning benchmarks, the model shows enhanced out-of-distribution generalization, with accuracy improvements of 4-6%.
- Training stability is notably improved, with RLCSD maintaining consistent response lengths and lower entropy fluctuations throughout training, avoiding the collapse or explosion phenomena observed in other OPSD baselines. Response length remains within optimal ranges, ensuring model robustness.
- Ablation studies reveal that the contrastive difference (ectr) reduces style bias by approximately 70%, as measured by style vs. task token attention metrics, leading to more content-focused outputs and higher reasoning accuracy. The method also demonstrates better cross-task transferability, indicating strong generalization.
Significance
This work addresses a critical bottleneck in training large reasoning models—style drift caused by privileged information conditioning. By introducing a contrastive mechanism, it effectively disentangles style from content, leading to more stable, content-focused models. The approach not only improves performance on reasoning benchmarks but also enhances the interpretability and robustness of models in real-world applications. It paves the way for more reliable AI systems capable of complex reasoning tasks, with potential impacts on education, scientific research, and automated decision-making. The integration of contrastive learning into reinforcement-based self-distillation opens new avenues for model training, especially in scenarios where access to teacher logits is limited or costly.
Technical Contribution
The main technical contributions include: • Introducing a contrastive token-level signal (ectr) that compares teacher distributions conditioned on correct and incorrect hints, effectively reducing stylistic bias. • Developing a symmetric sampling and subtraction scheme that leverages multiple negative hints to stabilize the contrastive signal. • Integrating the contrastive signal into a reinforcement learning framework via a two-path PPO, modulating the advantage estimate with a bounded, sign-preserving adjustment. • Demonstrating that this approach improves training stability, response length preservation, and reasoning accuracy across multiple large-scale models and tasks, outperforming existing OPSD methods.
Novelty
This research is the first to systematically incorporate contrastive learning into on-policy self-distillation for large language models, specifically targeting style drift caused by privileged conditioning. Unlike prior methods that rely solely on distribution divergence metrics, RLCSD constructs a symmetric contrastive signal that cancels shared stylistic components, focusing the learning on content-relevant tokens. This innovation bridges reinforcement learning, self-distillation, and contrastive learning, offering a new paradigm for content-focused model training.
Limitations
- The effectiveness of the contrastive signal heavily depends on the verifier's accuracy; inaccuracies can introduce noise, affecting the quality of the contrastive difference and model performance.
- In tasks with highly heterogeneous negative hints or complex multi-modal data, constructing stable and representative contrastive pairs remains challenging, potentially limiting scalability.
- Additional computational overhead from sampling multiple rollouts and negative hints increases training time and resource consumption, which may hinder deployment in resource-constrained environments.
Future Work
Future research should explore adaptive verifier mechanisms to improve robustness, extend the contrastive framework to multi-modal and multi-task settings, and optimize sampling strategies to reduce computational costs. Investigating the integration of this approach with other training paradigms, such as supervised fine-tuning and unsupervised learning, could further enhance model generalization. Additionally, applying the method to smaller models or real-world applications like dialogue systems and automated reasoning in scientific domains will be valuable.
AI Executive Summary
In the rapidly evolving field of large-scale reasoning models, a persistent challenge has been the tendency of models to drift stylistically when conditioned on privileged information, such as verified solutions or reference outputs. This phenomenon, termed style drift, manifests as overly concise, assertive, or stylistic outputs that often undermine the model’s reasoning depth and accuracy. Existing methods like reinforcement learning with verifiable rewards (RLVR) and on-policy distillation (OPSD) have made significant strides in improving reasoning capabilities, but they remain susceptible to style bias, leading to training instability and response length collapse.
This paper introduces RLCSD (Reinforcement Learning with Contrastive on-policy Self-Distillation), a novel framework designed to address these issues head-on. The core innovation lies in leveraging contrastive learning principles to construct a more content-focused, task-relevant training signal. By sampling multiple model rollouts and employing a verifier to classify them into correct and incorrect groups, the method constructs a contrastive token-level difference (ectr) that effectively cancels shared stylistic components. This contrastive signal is then integrated into a reinforcement learning setup, modulating the advantage estimate (AORM) through a two-path PPO mechanism. The result is a training process that is both more stable and more attentive to content relevance.
Extensive experiments on large models such as Qwen3 (1.7B, 4B, 8B) and Olmo-3-7B-Think across mathematical and logical reasoning tasks demonstrate the efficacy of RLCSD. The method consistently outperforms baseline approaches like GRPO, SDPO, and SRPO, achieving higher accuracy, improved training stability, and stable response lengths. Notably, the contrastive approach reduces style bias by approximately 70%, leading to more content-rich outputs and better generalization, especially in out-of-distribution scenarios.
The significance of this work extends beyond performance metrics. It provides a systematic solution to a fundamental problem in model training—style bias induced by privileged conditioning—thus enabling the development of more reliable, content-focused reasoning systems. The framework’s generality suggests it can be integrated into existing OPSD methods, offering a versatile tool for future research. While challenges remain, such as the dependence on verifier quality and computational overhead, the proposed approach marks a significant step toward more stable and effective large language models capable of complex reasoning tasks. Future directions include extending contrastive self-distillation to multi-modal data, refining verifier robustness, and scaling to even larger models, promising a broad impact on AI research and applications.
Deep Analysis
Background
The evolution of large language models (LLMs) has revolutionized natural language processing, enabling advanced reasoning, understanding, and generation capabilities. Early supervised fine-tuning methods, while effective, struggled with long-horizon reasoning due to sparse reward signals. Reinforcement learning with reward signals derived from external verifiers, such as RLHF and GRPO, addressed some issues but introduced new challenges, notably style bias—where models tend to produce shorter, more assertive, or stylistic outputs that deviate from content fidelity.
On-policy distillation (OPSD) emerged as a promising approach, leveraging the model’s own trajectories conditioned on privileged information (e.g., verified solutions) to provide dense, token-level supervision. This method accelerates convergence and reduces exposure bias, but it suffers from a systematic problem: the privileged conditioning causes the model to focus disproportionately on stylistic tokens—such as discourse markers or formatting cues—at the expense of task-relevant content. This phenomenon, termed style drift, leads to training instability, response length collapse, and reduced reasoning accuracy.
Recent efforts have attempted to mitigate style drift through heuristic techniques like KL-clipping or divergence balancing, but these are insufficient for large-scale, complex tasks. The need for a principled, scalable solution motivated the development of RLCSD, which integrates contrastive learning principles into the reinforcement learning framework to directly address the root cause of style bias.
Core Problem
The core problem addressed in this work is the phenomenon of privilege-induced style drift in on-policy self-distillation of large reasoning models. When models are conditioned on privileged hints—such as verified solutions—they tend to produce outputs that are stylistically shorter, more assertive, and less content-rich. This bias is driven by the shared stylistic components in the distribution conditioned on correct and incorrect hints, which dominate the token-level signals used for training. Consequently, the model's focus shifts away from content-bearing tokens like numerals, operators, or logical connectors, towards discourse markers or formatting tokens, leading to decreased reasoning depth, response length shrinkage, and training instability.
Existing methods, which rely solely on distribution divergence metrics like KL divergence, fail to disentangle style from content, resulting in persistent style drift. This not only hampers model performance on reasoning benchmarks but also reduces robustness and interpretability. The challenge is to develop a training mechanism that can suppress shared stylistic biases while emphasizing task-relevant content, ensuring stable training and improved reasoning accuracy.
Innovation
The key innovation of this paper is the introduction of a contrastive token-level signal (ectr) that compares the teacher's distribution conditioned on correct versus incorrect hints, effectively canceling shared stylistic components. This symmetric contrastive approach is designed to isolate content-related differences, making the training signal more task-specific. To implement this, the authors sample multiple negative hints from the incorrect group and average their teacher distributions, stabilizing the contrastive difference. They then normalize and modulate this signal using a bounded, sign-preserving mechanism, integrating it into a reinforcement learning framework via a two-path PPO setup.
This approach differs fundamentally from prior methods that rely solely on distribution divergence metrics, which often conflate style and content. By explicitly constructing a symmetric contrast, RLCSD effectively suppresses style bias, leading to more content-focused outputs. Additionally, the method employs a verifier to classify rollouts into correct and incorrect groups, ensuring the contrastive signals are grounded in task correctness. The integration of this contrastive mechanism into the advantage modulation process results in a training paradigm that is both more stable and more aligned with task content, addressing longstanding issues in large-scale reasoning model training.
Methodology
- �� Sampling multiple rollouts per query from the current policy, then using a verifier to classify each as correct or incorrect, forming G+ and G− groups.
- �� Constructing positive privileged contexts from correct rollouts and negative contexts from incorrect ones, both formatted with an identical prompt template to ensure stylistic consistency.
- �� Feeding these contexts into the teacher model to generate distributions conditioned on correct and incorrect hints, computing the distribution gaps ec,t and ew,t.
- �� Calculating the contrastive difference ectr,t = ec,t - ew,t, which cancels shared stylistic components, emphasizing task-relevant tokens.
- �� Applying a normalization (tanh) and scaling (λ) to ectr,t to produce a token-level modulation rt.
- �� Using a sign-preserving clamp to integrate rt into the advantage estimate (AORM), forming a two-path PPO loss with clipping and masking.
- �� Introducing negative hint marginalization (sampling multiple negatives) and excluding the current rollout from hint pools to improve stability.
- �� Updating the model parameters by optimizing the combined loss, which modulates the advantage with the contrastive signal, ensuring content focus and training stability.
Experiments
The experimental setup involves training large-scale models—Qwen3 (1.7B, 4B, 8B) and Olmo-3-7B-Think—on mathematical and logical reasoning tasks, including AMC23, AIME24, AIME25, and Knights & Knaves. The datasets include benchmark problems with verified solutions, and the evaluation metrics focus on pass@1 accuracy, response length stability, and training convergence. Baselines include GRPO, SDPO, SRPO, and RLSD. The experiments compare performance across different model sizes, task types, and out-of-distribution scenarios. Ablation studies examine the impact of the contrastive difference, negative hint sampling, and exclusion strategies. Hyperparameters such as λ, τ, and the number of negative hints (K) are tuned to optimize performance. The training process involves multiple epochs, with detailed logging of accuracy, response length, entropy, and style bias metrics.
Results
RLCSD consistently outperforms baseline methods across all models and tasks. For instance, in math reasoning, pass@1 accuracy on Qwen3-8B improves by 3-4%, reaching 88.8%, 74.2%, and 66.9%. Response length remains stable throughout training, avoiding collapse or explosion, unlike other OPSD baselines. Style bias metrics show a 70% reduction in stylistic token dominance, confirming the effectiveness of the contrastive signal. Out-of-distribution tests demonstrate enhanced robustness and generalization, with significant gains in logical reasoning benchmarks. Ablation results highlight the importance of negative hint sampling and symmetric contrast, with performance drops observed when these components are ablated.
Applications
The proposed method can be directly applied to improve reasoning systems in educational tools, scientific research assistants, and automated legal or medical analysis. Its ability to focus on content rather than style makes it suitable for tasks requiring high accuracy and interpretability. In industry, RLCSD can enhance chatbots, virtual assistants, and automated reasoning engines, especially in scenarios where model stability and content fidelity are critical. Long-term, integrating this contrastive self-distillation into multi-modal models could enable more robust AI systems capable of complex reasoning across diverse data types, fostering advancements in AI safety and reliability.
Limitations & Outlook
The approach relies heavily on the verifier's accuracy; inaccuracies can introduce noise into the contrastive signal, reducing effectiveness. Constructing effective negative hints in highly heterogeneous or multi-modal tasks remains challenging, potentially limiting scalability. The additional sampling and computation increase training costs, which may hinder deployment in resource-constrained environments. Moreover, the method's effectiveness on tasks beyond reasoning, such as dialogue generation or summarization, requires further validation. Future work should focus on verifier robustness, efficient sampling strategies, and extending the framework to broader AI applications.
Plain Language Accessible to non-experts
想象你在学校里学习写作文，老师（模型）希望你写得既有内容又有风格。有时候，你会模仿老师的写作风格，变得太过拘泥于格式或用词，反而忽略了作文的主题和内容。为了帮助你写得更好，老师会让你写两篇作文：一篇是按照正确的题目和内容写的，另一篇是写错了的版本。然后，老师会帮你比较这两篇作文，找出真正影响内容的差别，而不是只看风格上的不同。这样，你就能学会专注于写内容，而不是只追求漂亮的格式。这个方法就像在训练模型时，让它同时学习正确和错误的例子，通过对比，模型可以更清楚哪些部分是真正重要的内容，从而写出更准确、更合理的文章。最终，你的作文会变得更有内容，也更稳定，不会因为太在意风格而偏离主题。这就是用对比的方法，帮助模型专注于内容，避免偏向风格化输出的技巧。
ELI14 Explained like you're 14
想象你在学校参加写作比赛，你的老师（模型）希望你写得既有趣又有内容。有时候，你可能会用一些漂亮的词句，但内容却不够丰富，或者太过拘泥于格式，忽略了真正想表达的意思。为了帮你写得更好，老师会让你写两篇作文：一篇是按照题目和内容写的，另一篇是写错了的版本。然后，老师会帮你比较这两篇作文，找出哪些地方是真正表达了你的想法，哪些只是花哨的词句。这样，你就能学会专注于写内容，而不是只追求漂亮的外表。这个方法就像在训练模型一样，让它同时学习正确和错误的例子，通过对比，模型可以更清楚哪些部分是真正重要的内容，从而写出更合理、更有用的答案。最终，你的作文会变得更丰富、更有趣，也不会因为太在意外表而偏离主题。这就是用对比的方法，帮助模型专注于内容，而不是风格，让它变得更聪明、更可靠！
Abstract
On-policy self-distillation (OPSD) provides dense, token-level supervision for reasoning models by aligning a model's own distribution with the distribution it produces under privileged context, typically a verified solution. However, we show that the learning signal drawn from this distributional gap concentrates on style tokens rather than task-bearing ones, as the hinted model tends to produce more direct, shorter outputs. We term this pathology \emph{privilege-induced style drift}, which destabilizes training or causes response length to shrink. To address this, we propose \textbf{RLCSD} (Reinforcement Learning with Contrastive on-policy Self-Distillation), which mitigates this drift by contrasting the teacher-student gap under a correct hint against that under a wrong hint, suppressing the style shift that conditioning on a hint tends to induce regardless of correctness, and yielding a signal that is more concentrated on task-bearing tokens. Experiments on Qwen3 (1.7B/4B/8B) and Olmo-3-7B-Think across mathematical and logical reasoning show that RLCSD consistently outperforms GRPO and prior OPSD methods. We further show that the contrastive principle is general: it plugs into existing OPSD methods to improve them, and its underlying insight extends to the broader cross-model on-policy distillation setting.
References (20)
MiMo-V2-Flash Technical Report
Xi Xiao, Bing Xia, Bo Yang et al.
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
Adam Suma, Sam Dauncey
Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
Siyan Zhao, Zhihui Xie, Mengchen Liu et al.
Reinforcement Learning via Self-Distillation
Jonas Hubotter, Frederike Lubeck, L. Behric et al.
On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou et al.
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
Zhihong Shao, Peiyi Wang, Qihao Zhu et al.
UniSD: Towards a Unified Self-Distillation Framework for Large Language Models
Yiqiao Jin, Yiyang Wang, Lucheng Fu et al.
Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing
Gengsheng Li, Tianyu Yang, Junfeng Fang et al.
Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning
Tian Xie, Zitian Gao, Qingnan Ren et al.
Black-Box On-Policy Distillation of Large Language Models
Tianzhu Ye, Li Dong, Zewen Chi et al.
Group Sequence Policy Optimization
Chujie Zheng, Shixuan Liu, Mingze Li et al.
GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning
GLM-V Team Wenyi Hong, Wenmeng Yu, Xiaotao Gu et al.
DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning
Zhiwei He, Tian Liang, Jiahao Xu et al.
Understanding R1-Zero-Like Training: A Critical Perspective
Zi-Yan Liu, Changyu Chen, Wenjun Li et al.
DAPO: An Open-Source LLM Reinforcement Learning System at Scale
Qiying Yu, Zheng Zhang, Ruofei Zhu et al.
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Ronald J. Williams
HybridFlow: A Flexible and Efficient RLHF Framework
Guangming Sheng, Chi Zhang, Zilingfeng Ye et al.
DistiLLM: Towards Streamlined Distillation for Large Language Models
Jongwoo Ko, Sungnyun Kim, Tianyi Chen et al.
Cited By (15)
Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation
Trust Is Not Enough: Influence Calibration for On-Policy Self-Distillation in Agentic RL
Latent On-Policy Self-Distillation
Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillation
Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents
Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation
Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation
SAF-OPD: Stable Advantage Fusion for On-Policy Distillation
Not All Tokens Deserve Equal Credit: Counterfactual Sensitivity Credit Reallocation for Long-CoT Reasoning
Contrastive Reinforced Policy Optimization via Privileged Self-Distillation
Group-Reflective Self-Distillation for Agentic Reinforcement Learning
Weak-to-Strong On-Policy Distillation
CriPO: Enhancing Rubric-based RL via Self-Distillation
Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops
DRIFT: Difficulty Routing Self-DIstillation with Rhythm-Gated Exploration and Success BuFfer Training