Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning
GC-OPD introduces group-normalized verifier rewards and relative advantage credit assignment, boosting long-context reasoning scores from 29.08 to 40.47 on five benchmarks.
Key Findings
Methodology
This paper proposes Group-Calibrated On-Policy Distillation (GC-OPD), which enhances traditional OPD by normalizing verifier rewards and trajectory-level OPD scores within each rollout group to form a signed residual indicating teacher-verifier disagreement. The residual captures the direction and magnitude of bias between teacher preferences and task-specific verifier assessments. The approach employs Relative-Advantage-based Credit Assignment (RACA), distributing this residual across tokens based on their relative OPD advantages, thus maintaining dense token-level guidance while correcting response-level biases. Extensive experiments across five long-context benchmarks demonstrate that GC-OPD significantly outperforms vanilla OPD, with improvements from 29.08 to 40.47 (Qwen3-4B) and from 35.12 to 44.65 (Qwen3-8B). The ablation studies confirm the effectiveness of the residual and RACA, establishing a new state-of-the-art in long-context reasoning tasks.
Key Results
- In five long-context benchmarks, GC-OPD raised the average score of Qwen3-4B from 29.08 to 40.47, an improvement of approximately 11.39 points, and for Qwen3-8B from 35.12 to 44.65, an increase of 9.53 points. The performance gains are especially prominent in structured reasoning and evidence aggregation tasks such as DocMath, MRCR, and CorpusQA, indicating that bias correction at the response level effectively enhances global reasoning capabilities.
- Ablation experiments show that the signed residual derived from group-normalized verifier rewards outperforms simply adding extra OPD terms or direct reward addition. RACA further improves token allocation over uniform distribution, leading to more precise bias correction. These results validate the importance of dynamic, relative advantage-based residual distribution in long-context settings.
- Across different model scales, including Qwen3-4B and Qwen3-8B, GC-OPD consistently achieves the highest average scores, demonstrating its robustness and general applicability. The method’s effectiveness is confirmed across diverse tasks, including numerical reasoning, multi-hop synthesis, and long-document summarization.
Significance
This work addresses a fundamental challenge in long-context language modeling: the mismatch between dense teacher guidance and task-level verifier feedback. By introducing a group-relative residual calibration mechanism, the method effectively aligns token-level supervision with global task success signals. This advances the state-of-the-art in long-context reasoning, enabling models to better integrate dispersed evidence and adhere to global constraints. The approach offers a scalable, modular solution that can be integrated into existing distillation pipelines, with broad implications for AI applications requiring deep understanding and reasoning over extended texts.
Technical Contribution
The paper’s key technical innovation lies in the group-normalized residual calibration framework, which captures teacher-verifier disagreement at the group level and propagates it through token-level adjustments via RACA. This preserves dense token guidance while correcting for global biases, providing a theoretically sound and empirically validated mechanism for bias mitigation. The method introduces minimal additional computational overhead, as it leverages existing token advantages and verifier scores, making it practical for large-scale training. The combination of normalization, residual formulation, and relative advantage-based distribution constitutes a novel contribution to bias correction in reinforcement learning and distillation settings.
Novelty
This is the first work to explicitly model teacher-verifier disagreement through group-normalized residuals and to distribute this residual adaptively across tokens using relative advantages. Unlike prior approaches that treat verifier feedback as an auxiliary signal or apply simple reward adjustments, GC-OPD dynamically calibrates biases at the token level, maintaining dense supervision. Its integration of group-relative normalization with advantage-based residual distribution represents a significant conceptual and technical advancement in bias correction for long-context models, setting a new paradigm for combining local guidance with global task success signals.
Limitations
- The effectiveness of GC-OPD heavily depends on the quality and design of the verifier. Poorly calibrated or biased verifier models may introduce misleading residuals, potentially degrading performance.
- Normalization within groups assumes that the distribution of verifier rewards and OPD scores is stable; in highly skewed or noisy scenarios, this assumption may break down, leading to unstable training.
- The additional computational overhead, although minimal, still poses challenges for extremely large-scale training or real-time inference scenarios. Further optimization is needed for deployment in resource-constrained environments.
Future Work
Future research could explore adaptive normalization strategies that better handle skewed or noisy data, as well as integrating multi-modal verification signals. Extending the residual calibration framework to multi-task and multi-modal settings, including vision-language tasks, could broaden its applicability. Additionally, developing more robust, learnable verifier models and exploring unsupervised or self-supervised bias correction methods are promising directions to further enhance long-context reasoning capabilities.
AI Executive Summary
The rapid development of large-scale language models has revolutionized natural language understanding and generation. However, their performance in long-context reasoning tasks remains limited by biases introduced during training, especially when relying solely on teacher models for guidance. Traditional knowledge distillation approaches like On-Policy Distillation (OPD) leverage dense token-level advantages from a strong teacher to improve response quality. Yet, in long-input scenarios, this dense guidance often favors responses that are locally plausible but globally incomplete or inconsistent with task-specific constraints. This discrepancy between teacher preference and task success signals hampers the model’s ability to reason over extended contexts effectively.
To address this, the authors introduce Group-Calibrated On-Policy Distillation (GC-OPD), a novel framework that explicitly models and corrects teacher-verifier disagreement. The core idea is to normalize both verifier rewards and trajectory-level OPD scores within each rollout group, forming a signed residual that indicates the direction and magnitude of bias. This residual captures whether the teacher’s preference aligns with the task verification, enabling targeted bias correction. The residual is then distributed across tokens using Relative-Advantage-based Credit Assignment (RACA), which allocates bias correction proportionally to each token’s relative OPD advantage, preserving the dense token-level guidance.
Experimental results across five long-context benchmarks demonstrate the effectiveness of GC-OPD. The method consistently outperforms vanilla OPD, with average score improvements of over 10 points on both Qwen3-4B and Qwen3-8B models. Notably, the gains are most significant in tasks involving evidence aggregation and structured reasoning, validating the hypothesis that bias correction at the response level enhances global understanding. Ablation studies confirm that the signed residual and RACA are crucial components, with each contributing to performance gains.
This work has broad implications for advancing long-context reasoning in large language models. By effectively integrating task-specific verifier feedback with dense token guidance, GC-OPD offers a scalable, modular solution to long-standing challenges in bias and bias correction. Its principles can be extended to multi-modal, multi-task, and real-world applications, paving the way for more reliable and globally consistent AI systems. Despite some limitations related to verifier quality and computational costs, the framework sets a new benchmark for bias-aware distillation in long-context NLP tasks, promising significant impact in both academia and industry.
Deep Dive
Abstract
On-policy distillation (OPD) trains a student on its own responses using dense token-level guidance from a stronger teacher. In long-context tasks, however, token-level teacher support can favor locally plausible responses that omit evidence distributed across the input or violate global task constraints. Task-specific verifiers, in contrast, evaluate task completion at the response level and may return graded rewards that reflect partial success. We diagnose this mismatch on fixed responses from two representative long-context evidence-aggregation tasks. Across longer input ranges, trajectory-level OPD scores become progressively less aligned with verifier rewards, indicating teacher-verifier disagreement. Motivated by this observation, we introduce Group-Calibrated On-Policy Distillation (GC-OPD). GC-OPD separately normalizes verifier rewards and trajectory-level OPD scores within each rollout group and uses their difference as a signed teacher-verifier disagreement residual. Relative-advantage-based credit assignment (RACA) distributes this trajectory-level residual across tokens according to their relative OPD advantages while preserving the original OPD signal. Across five long-context benchmarks, post-training with GC-OPD raises the five-benchmark averages of the official Qwen3-4B and Qwen3-8B checkpoints from 29.08 to 40.47 and from 35.12 to 44.65, respectively. Vanilla OPD reaches 39.31 and 43.56 under the same setup. Controlled ablations show that the signed residual is more effective than either an additional OPD-derived term or direct group-normalized verifier reward addition, while RACA further improves over uniform token allocation. Together, these results demonstrate that group-relative residual calibration can incorporate verifier outcomes without discarding dense token-level guidance. Code is available at https://github.com/SolereZhang/GC-OPD.
References (20)
PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation
Anhao Zhao, Junlong Tong, Yingqi Fan et al.
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
Zhihong Shao, Peiyi Wang, Qihao Zhu et al.
On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou et al.
QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learning
Fanqi Wan, Weizhou Shen, Shengyi Liao et al.
Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries
Kiran Vodrahalli, Santiago Ontañón, Nilesh Tripuraneni et al.
VinePPO: Refining Credit Assignment in RL Training of LLMs
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance et al.
RULER: What's the Real Context Size of Your Long-Context Language Models?
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman et al.
Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation
Wenkai Yang, Weijie Liu, Ruobing Xie et al.
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
Yushi Bai, Xin Lv, Jiajie Zhang et al.
Let's Verify Step by Step
H. Lightman, Vineet Kosaraju, Yura Burda et al.
Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe
Wenjin Hou, Shangpin Peng, Weinong Wang et al.
Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLP
Omer Goldman, Alon Jacovi, Aviv Slobodkin et al.
GoLongRL: Capability-Oriented Long Context Reinforcement Learning with Multitask Alignment
Minxuan Lv, Tiehua Mei, Tanlong Du et al.
SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling
Haoran Xu, Hongyu Wang, Yifei Gao et al.
DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documents
Yilun Zhao, Yitao Long, Hongjun Liu et al.
Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillation
Minsang Kim, S. Baek
LongAlign: A Recipe for Long Context Alignment of Large Language Models
Yushi Bai, Xin Lv, Jiajie Zhang et al.
Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation
Yuanyi Wang, Su Lu, Yanggan Gu et al.
DistiLLM: Towards Streamlined Distillation for Large Language Models
Jongwoo Ko, Sungnyun Kim, Tianyi Chen et al.