Contrastive On-Policy Distillation
COPD enhances student model reasoning efficiency by using contrastive teaching signals to reduce reasoning length.
Key Findings
Methodology
COPD uses contrastive teaching signals to provide comparative information between light and heavy reasoning, guiding student model optimization. A frozen teacher model scores the same student state under both light and heavy reasoning, with the difference serving as a signal for updates.
Key Results
- COPD reduces reasoning length and improves efficiency across nine multimodal benchmarks, increasing Acc@1K by 141.2%.
- Compared to ExOPD, COPD improves accuracy by 11.3 percentage points on WeMath while reducing reasoning length by 60.7%.
- The COPSD variant demonstrates self-distillation capabilities without external teachers.
Significance
COPD enhances reasoning efficiency by reducing redundant reasoning steps, applicable to various tasks and model scales. It holds significant academic and industrial implications by improving computational efficiency.
Technical Contribution
COPD replaces single-distribution imitation with contrastive signals, offering a new method for reasoning compression. Unlike existing methods, COPD emphasizes relative preference between reasoning modes, providing a more effective learning signal.
Novelty
COPD is the first to introduce contrastive signals into policy distillation, emphasizing relative preference between reasoning modes rather than single-distribution imitation, significantly enhancing reasoning efficiency.
Limitations
- COPD excels in reasoning tasks but may reduce accuracy in tasks with very short outputs.
- The effectiveness of contrastive signals depends on the frozen state of the teacher model.
Future Work
Future research can explore COPD's application to more tasks and optimize the generation mechanism of contrastive signals to enhance model adaptability.
AI Executive Summary
In recent years, as large-scale reasoning models have advanced, redundancy in the reasoning process has become increasingly prominent. Existing on-policy distillation methods rely on single-distribution imitation, lacking modeling of relative preference between reasoning modes.
COPD introduces contrastive signals to provide comparative information between light and heavy reasoning, guiding student model optimization. A frozen teacher model scores the same student state under both light and heavy reasoning, with the difference serving as a signal for updates. COPD not only reduces reasoning length but also improves reasoning efficiency.
Experimental results show that COPD performs exceptionally well across nine multimodal benchmarks, significantly reducing reasoning length without compromising performance. The COPSD variant further demonstrates self-distillation capabilities without external teachers. Future research can explore COPD's application to more tasks and optimize the generation mechanism of contrastive signals.
Deep Analysis
Background
In recent years, reasoning models have made significant progress in multimodal understanding and text reasoning tasks. However, as reasoning capabilities improve, redundancy in the reasoning process has become increasingly prominent. Existing methods primarily enhance efficiency through selected concise traces, preference pairs, token budgets, or reward mechanisms.
Core Problem
Existing on-policy distillation methods rely on single-distribution imitation, lacking modeling of relative preference between reasoning modes. This results in a lack of explicit comparison signals between reasoning modes, preventing direct modeling of preferences between these modes.
Innovation
COPD introduces contrastive signals to provide comparative information between light and heavy reasoning, guiding student model optimization. A frozen teacher model scores the same student state under both light and heavy reasoning, with the difference serving as a signal for updates.
Methodology
- �� Student model generates reasoning trajectories.
- �� Frozen teacher model scores the same trajectory under light and heavy reasoning.
- �� Compute the difference as a contrastive signal.
- �� Use the contrastive signal to guide student model updates.
Experiments
Experiments are conducted on nine multimodal benchmarks, including reasoning and understanding tasks. COPD significantly reduces reasoning length and improves efficiency across these tasks. The experiments use models like Qwen3-VL-2B-Instruct and Qwen3.5-2B.
Results
COPD reduces reasoning length and improves reasoning efficiency across all tasks. Compared to ExOPD, COPD improves accuracy by 11.3 percentage points on WeMath while reducing reasoning length by 60.7%.
Applications
COPD is applicable to multimodal tasks requiring efficient reasoning, such as mathematical problem solving, logical inference, and multi-step decision making.
Limitations & Outlook
COPD excels in reasoning tasks but may reduce accuracy in tasks with very short outputs. The effectiveness of contrastive signals depends on the frozen state of the teacher model.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. Existing methods are like following a recipe step by step, while COPD is like having an experienced chef beside you, guiding you on which steps can be simplified and which need careful attention. Through this contrastive guidance, you can cook faster without compromising taste.
ELI14 Explained like you're 14
Imagine you're playing a puzzle game. Usually, you'd try every possible solution step by step, but that can be time-consuming. COPD is like the game's hint system, telling you which steps you can skip and which ones need careful consideration. This way, you can finish the game faster without losing the fun!
Glossary
Contrastive Learning
A technique that learns features by comparing similarities and differences between samples.
COPD uses contrastive learning to guide student model optimization.
On-Policy Distillation
A technique that supervises on trajectories generated by the student model.
COPD provides contrastive signals in on-policy distillation.
Multimodal
Involving multiple data types, such as images and text.
COPD is tested on multimodal benchmarks.
Self-distillation
A distillation technique without an external teacher model.
The COPSD variant demonstrates self-distillation capabilities.
Reasoning Efficiency
The ability to reduce reasoning steps while maintaining performance.
COPD significantly improves reasoning efficiency.
Open Questions Unanswered questions from this research
- 1 How to maintain COPD's effectiveness in tasks with very short outputs?
- 2 How to optimize the generation mechanism of contrastive signals to enhance model adaptability?
Applications
Immediate Applications
Mathematical Problem Solving
COPD can be used to improve the efficiency of solving mathematical problems by reducing unnecessary reasoning steps.
Long-term Vision
General AI Systems
COPD's contrastive signal mechanism can be used to build more efficient general AI systems, reducing computational costs.
Abstract
On-policy Distillation (OPD) supervises a student model on trajectories sampled from its own policy by minimizing the divergence between the output distributions of the teacher and student at each token position, thereby providing dense token-level supervision. Although existing OPD methods have demonstrated strong performance in improving the reasoning ability of student models, their objectives fundamentally rely on token-level distribution matching. Consequently, they lack an explicit signal for comparing a token's relative compatibility across reasoning modes and thus do not directly model preferences between these modes. To address this limitation, we propose COPD, a contrastive OPD framework. Specifically, for each token generated by the student model, a frozen teacher model scores the same student state under two contrasting instructions that elicit light and heavy reasoning. The difference between the resulting log probabilities serves as a token-level advantage signal to guide the OPD update. Rather than merely imitating a single teacher distribution, COPD directly encourages the student model to learn more concise and efficient reasoning strategies. We conduct experiments on nine multimodal benchmarks covering both reasoning and understanding tasks. The results show that COPD substantially reduces reasoning length without compromising model performance and consistently improves efficiency across different tasks and model scales. Furthermore, the contrastive formulation can be seamlessly integrated into the On-policy Self-distillation (OPSD) framework, where self-contrastive supervision is constructed without an additional teacher model, thereby enabling the model to distill itself toward lightweight reasoning.