SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning
SimpleOPD enables cross-tokenizer long-context reasoning transfer, boosting mathematical proof performance by 21.2 points on ProofBench.
Key Findings
Methodology
This paper introduces SimpleOPD, a tokenizer-agnostic on-policy distillation framework that aligns responses in a shared text space. It employs token span matching to handle tokenizer mismatch, integrates a student reference KL loss to stabilize training, and masks special termination tokens to prevent length explosion. The approach uses PPO-based objectives to optimize response alignment, effectively transferring reasoning capabilities from long-context teachers like SU-01 to diverse short-context student models, including models from Qwen, Intern, GLM, and Gemma families. Extensive experiments demonstrate significant improvements in mathematical reasoning and scientific benchmarks, especially in cross-model and cross-tokenizer settings.
Key Results
- On ProofBench, Intern-S2-Preview improved by 21.2 points, reaching 55.2, surpassing Gemini-2.5-Pro, demonstrating effective transfer of long-context reasoning.
- Across science benchmarks HLE and HiPhO, the distilled models showed notable gains, indicating generalization beyond mathematical domains.
- Incorporating KL regularization and termination token masking enhanced training stability, controlled response length, and improved overall performance.
Significance
This work addresses critical limitations in transferring reasoning skills across different model architectures and tokenization schemes. By enabling tokenizer-agnostic alignment, it broadens the applicability of model distillation for complex reasoning tasks. The approach enhances the generalization and robustness of smaller models, making advanced reasoning capabilities accessible in resource-constrained settings. Its success in scientific and mathematical benchmarks underscores its potential for real-world applications in education, research, and AI-powered problem solving, marking a significant step forward in scalable reasoning transfer.
Technical Contribution
The core technical innovation lies in the shared text space alignment mechanism that bypasses tokenization incompatibility, combined with a student reference KL loss that regularizes policy drift. Masking special termination tokens prevents response length runaway, while PPO-based optimization ensures stable policy updates. These components collectively enable effective long-context reasoning transfer across diverse models and tokenizers, setting a new standard for tokenizer-agnostic distillation frameworks.
Novelty
This is the first work to propose a tokenizer-agnostic, cross-model long-context reasoning transfer method that does not rely on shared tokenization schemes. Its novel alignment in shared text space and the combined use of KL regularization and termination masking distinguish it from prior works limited to same-tokenizer or off-policy distillation, opening new avenues for flexible, scalable knowledge transfer.
Limitations
- The method still faces challenges in extremely long contexts or multi-modal scenarios where response length control and alignment become more complex.
- Training costs are high due to the need for multiple rollout iterations and large models, limiting immediate scalability.
- Alignment based on text span matching may struggle with multilingual or highly noisy data, requiring further robustness improvements.
Future Work
Future research will explore multi-modal extensions, adaptive length regulation, and unsupervised alignment techniques to further improve stability and efficiency. Additionally, integrating this framework with multi-task learning and continual training could expand its applicability, enabling more versatile and resource-efficient reasoning transfer in diverse AI systems.
AI Executive Summary
The rapid development of large language models (LLMs) has revolutionized natural language understanding and generation. However, transferring complex reasoning skills, especially long-context mathematical proofs, remains a challenge, particularly when models differ in architecture and tokenization schemes. Traditional distillation methods often require identical tokenizers and architectures, limiting their scalability and applicability across diverse models. This paper introduces SimpleOPD, a novel tokenizer-agnostic on-policy distillation approach designed to bridge this gap.
At its core, SimpleOPD performs response alignment in a shared text space, matching tokens based on identical text spans rather than token indices. This innovative alignment mechanism effectively handles tokenizer mismatch, enabling knowledge transfer across models with different vocabularies and tokenization rules. To further stabilize training, the authors incorporate a student reference KL loss, which constrains the student model's policy from drifting excessively from its initial distribution. Additionally, masking special termination tokens like </think> and <|im_end|> prevents the model from generating excessively long responses, addressing a common problem in long-context training.
Experimental results demonstrate the method’s effectiveness. When applied to models like Intern-S2-Preview, the approach yields a 21.2-point improvement on ProofBench, reaching 55.2, surpassing some state-of-the-art models. The models also show enhanced performance on science benchmarks such as HLE and HiPhO, indicating that the transferred reasoning capabilities are not domain-specific but generalize broadly. The combination of shared text alignment, KL regularization, and termination masking results in a stable training process, with responses tightly controlled in length and quality.
This work significantly advances the field of model distillation, especially for complex reasoning tasks requiring long contexts. It opens new avenues for deploying smaller, resource-efficient models with reasoning skills comparable to larger teachers, across diverse architectures and tokenization schemes. The approach’s flexibility and robustness suggest promising future directions, including multi-modal reasoning, multi-task transfer, and adaptive response length control. Overall, SimpleOPD marks a pivotal step toward scalable, generalizable AI reasoning systems.
Deep Analysis
Background
近年来，深度学习模型在自然语言处理中的表现持续突破，尤其是大型预训练模型（如GPT、BERT）在多任务、多领域展现出强大能力。模型蒸馏作为提升推理和生成能力的重要手段，已成为研究热点。传统蒸馏方法多依赖于相同分词器和模型架构，限制了跨模型迁移的效果。长上下文推理能力，尤其在数学证明和科学推理中，仍面临分词器不一致、响应长度控制和训练不稳定等挑战。现有的Off-policy蒸馏在复杂推理任务中的应用效果有限，难以满足实际需求。
Core Problem
核心问题在于如何在不同分词器和模型架构间实现高效、稳定的长上下文推理迁移。具体表现为：分词器不一致导致Token对齐困难，模型容量差异引起分布偏差，长响应导致训练不稳定和响应爆炸。此外，终止Token的对齐和响应长度控制成为难点，影响迁移效果和模型泛化能力。这些问题限制了模型在复杂推理任务中的应用，亟需创新的解决方案。
Innovation
本研究提出三大创新：1）在共享文本空间中对齐响应文本跨度，解决不同分词器的对齐难题；2）引入学生参考KL正则化，稳定训练过程，防止响应长度爆炸；3）屏蔽特殊终止Token的优势，避免模型过度偏离初始策略。这些创新结合PPO优化框架，有效缓解教师-学生分布差异，提升迁移效率，显著优于现有方法。
Methodology
- �� 构建共享文本空间，将教师和学生的响应在相同文本跨度上对齐。• 通过响应文本匹配机制，定义Token对应关系，解决不同分词器引起的对齐问题。• 采用基于PPO的对齐目标，只在匹配Token位置上进行监督，减少偏差。• 引入学生参考KL正则化，限制模型偏离初始策略，控制响应长度。• 屏蔽终止Token的优势，避免响应无限膨胀。• 使用响应匹配的对齐策略，确保迁移过程中的稳定性和效果。
Experiments
在数学证明和科学任务中，采用SU-01作为教师模型，迁移到Qwen3、Qwen3.5、Intern-S2等多模型。训练数据包括OPC、AoPS、书籍和社区题库。采用PPO优化，训练100轮，批次64，响应长度上限32K。评估指标包括ProofBench、AnswerBench、AIME25等，采用多次评估取平均。对比传统蒸馏和本方法，验证稳定性和性能提升。
Results
在ProofBench上，Intern-S2-Preview由34.0提升至55.2，增长21.2分，超越Gemini-2.5-Pro。科学任务HLE和HiPhO也获得显著提升，验证推理迁移的有效性。引入KL正则化和终止Token屏蔽后，训练更稳定，响应长度受控，模型表现优异。多模型迁移实验显示，方法具有良好的泛化能力和适应性。
Applications
该技术适用于教育、科研、智能问答等场景，尤其适合需要长上下文推理的复杂任务。通过无监督对齐和稳定训练，降低模型迁移门槛，提升实际应用表现。未来可结合多模态信息，拓展到多语言、多任务环境，推动智能系统的智能化升级。
Limitations & Outlook
目前方法在极端长上下文或多模态场景下仍存在响应长度控制不足的问题。训练成本较高，尤其在大模型迁移中，需优化效率。对齐机制依赖文本跨度匹配，可能在多语言或多模态场景中面临挑战。未来需探索更高效的长度调节和多模态适应策略。
Plain Language Accessible to non-experts
想象你在教一个学生做数学题。老师（模型）非常厉害，能解决复杂问题，但他用一种特殊的语言（分词器）表达答案。你（学生）用另一种语言，但你们都在看同一张题目纸。为了让你学会老师的解题方法，老师会用一种特殊的方式告诉你哪些部分是关键，哪些可以跳过。你们会一起对照答案的不同部分，找到对应的内容。通过不断练习，你逐渐学会了老师的思路，即使你用不同的语言表达，也能理解和解决问题。这就像本文的方法，让不同模型用不同“语言”也能学会长篇推理，变得更聪明、更稳定。
ELI14 Explained like you're 14
想象你在学校里学习一门超级难的数学课。老师讲得很快，内容也很长，有时候你听不完就得停下来休息。以前的老师（模型）都用一样的语言和方法讲题，这样你们交流就很顺畅。但现在，有个新老师用不同的语言（分词器），你听不懂，怎么学会他的推理呢？这篇文章就像发明了一种新办法，让你用自己的语言理解老师的讲解，还能找到老师讲的重点。这样，不管老师用什么语言，你都能学会长篇复杂的推理，不再怕内容太长或表达不一样了。这让学习变得更容易，也让你变得更聪明！
Abstract
On-policy distillation (OPD) offers a promising way to transfer reasoning capabilities from stronger teacher models, but applying it to long-context reasoning teachers and short-context students introduces practical challenges, including tokenizer mismatch, teacher-student distribution mismatch, response length explosion, and training instability. In this work, we study this setting by transferring proof-reasoning capabilities from the long-context reasoning model SU-01 to short-context student models. To handle tokenizer differences, we perform OPD in a shared text space and align only tokens that occupy identical text spans under the student and teacher tokenizers. To mitigate the problem of excessive generation length and frequent truncation, we introduce a student reference KL loss and mask the advantages of special termination tokens such as </think> and <|im_end|>. This strategy constrains the student from drifting excessively from its initial policy, thereby mitigating the teacher-student distribution mismatch problem and fostering steady length growth. Experiments on both same-family and different-family student models, including Qwen3, Qwen3.5, Intern-S2, GLM-4.7, Gemma-4, show consistent gains in mathematical reasoning, especially natural-language math proving. Notably, Intern-S2-Preview improves by 21.2 points on ProofBench, reaching 55.2 and surpassing Gemini-2.5-Pro. It also improves on science benchmarks such as HLE and HiPhO, suggesting that OPD transfers reasoning capabilities that generalize beyond the mathematical training domain.