Latent On-Policy Self-Distillation
LOPD introduces learnable latent privileged context, outperforming OPSD variants with less than 30% rollout budget.
Key Findings
Methodology
LOPD retrieves relevant experiences and encodes them into continuous latent tokens, conditioning a self-teacher. The student generates trajectories from task and interaction history, receiving dense token-level supervision via a differentiable latent context. The framework employs retrieval, encoding, compression, and joint optimization with reverse-KL distillation and privileged-margin constraints. The teacher conditions on learnable latent tokens, which are optimized end-to-end, enabling automatic extraction of task-relevant knowledge. The training process involves sampling trajectories, retrieving experiences, encoding into latent tokens, constructing context, evaluating with the fixed teacher, and jointly updating parameters to maximize supervision quality while maintaining a privilege margin.
Key Results
- LOPD surpasses RLVR and OPSD methods across multiple benchmarks, with performance gains of 8-12 points on QWEN3-8B models, reaching 62.7 on ACEBench with less than 30% of the rollout budget used by baselines. It demonstrates superior transferability across tasks and models, with consistent improvements in tool use and code generation domains.
- In experiments on QWEN3-4B and OLMO3-7B, LOPD outperforms baselines like Skill-SD and SDPO, confirming the effectiveness of learned latent context. Ablation studies show that end-to-end learning of the privileged context is crucial for these gains, with the privileged-margin constraint preventing trivial solutions.
- Overall, LOPD achieves state-of-the-art results, validating that adaptive, learnable privileged context significantly enhances self-distillation, especially under limited rollout budgets, and generalizes well across tasks and model sizes.
Significance
This work addresses the core limitation of fixed, hand-crafted privileged contexts in on-policy self-distillation, proposing a paradigm shift toward learnable, adaptive supervision substrates. By enabling models to autonomously extract and organize experience information, LOPD enhances scalability, efficiency, and robustness of self-evolving AI systems. Its ability to outperform existing methods on diverse benchmarks demonstrates its potential to accelerate AI's autonomous learning capabilities, paving the way for more scalable, self-directed agents capable of continual improvement without manual intervention. This approach opens new avenues for research in adaptive supervision, experience organization, and scalable AI development.
Technical Contribution
LOPD's main technical innovation lies in integrating retrieval, encoding, compression, and joint optimization of a learnable latent privileged context within the on-policy distillation framework. The method employs a differentiable composer to encode retrieved experiences into continuous tokens, conditioned on which a fixed backbone teacher re-evaluates prefixes, producing dense supervision signals. The framework introduces a privileged-margin constraint to prevent trivial solutions, and employs reverse-KL distillation for effective policy learning. This end-to-end trainable setup allows the model to automatically learn which experience aspects are most beneficial, significantly advancing the state-of-the-art in adaptive, scalable self-distillation.
Novelty
LOPD is the first to treat privileged context as a learnable latent representation, replacing fixed, rule-based artifacts. Unlike prior OPSD methods that rely on handcrafted artifacts like answers or trajectories, LOPD dynamically encodes experience into continuous tokens, enabling the model to autonomously determine what knowledge to retain and how to organize it. This innovation fundamentally shifts the paradigm from static supervision to adaptive, experience-driven learning, opening new directions for scalable self-evolving AI systems.
Limitations
- The effectiveness of LOPD heavily depends on the quality of experience retrieval; poor retrieval results can limit the richness and relevance of the latent context, affecting overall performance.
- The encoding and compression mechanisms may not fully capture all task-critical information, especially in highly complex or long-horizon tasks, potentially limiting scalability.
- Training involves complex joint optimization with multiple constraints, which can lead to instability or slow convergence, requiring careful hyperparameter tuning and computational resources.
Future Work
Future research could explore richer experience sources, including multimodal data, to enhance context expressiveness. Incorporating meta-learning or adaptive mechanisms may improve generalization to unseen tasks. Additionally, optimizing retrieval strategies and compression architectures could reduce training costs and improve stability. Extending the framework to real-world applications, such as robotics or autonomous systems, remains a promising direction for making self-evolving AI more practical and robust.
AI Executive Summary
The quest for autonomous, continually improving AI agents has driven significant research into self-distillation techniques. Traditional on-policy self-distillation (OPSD) methods rely heavily on hand-crafted privileged contexts—such as verified answers, reasoning traces, or expert trajectories—to provide dense supervision signals. While effective, these fixed artifacts limit the scalability and adaptability of the models, constraining their ability to learn from experience in a fully autonomous manner.
This paper introduces Latent On-Policy Self-Distillation (LOPD), a novel framework that fundamentally rethinks the role of privileged context. Instead of relying on manually designed artifacts, LOPD makes the privileged context itself learnable and adaptive. It retrieves relevant past experiences, encodes them into continuous latent tokens via a differentiable compressor, and conditions a self-teacher on this learned context. The student generates trajectories based on current task and interaction history, receiving dense supervision at every step. The entire process is optimized end-to-end, with a privileged-margin constraint ensuring the teacher remains informative and prevents collapse.
Experimental results across multiple benchmarks and model sizes demonstrate LOPD’s superiority. On tool use and code generation tasks, it outperforms existing methods like RLVR, SDPO, and Skill-SD, achieving performance improvements of up to 12 points and reaching scores such as 62.7 on ACEBench with less than 30% of the baseline rollout budget. The approach generalizes across diverse tasks and models, validating the importance of learnable privileged context for scalable self-evolution.
LOPD’s innovations include the integration of retrieval, continuous encoding, and joint optimization within a unified framework, enabling models to autonomously determine which experience aspects are most relevant. This work addresses key limitations of fixed, rule-based supervision, opening new avenues for scalable, self-directed AI systems capable of continual learning and adaptation. Its implications span robotics, programming, and multimodal AI, promising a future where AI agents learn more like humans—by autonomously organizing and internalizing their experiences—rather than relying solely on static datasets or handcrafted rules.
Deep Dive
Plain Language Accessible to non-experts
想象你在学习做菜。每次你尝试做一道菜，你会记住用的材料、步骤、遇到的问题。传统的方法就像老师提前告诉你一份详细的菜谱，你照着做。而LOPD则像是你自己在厨房不断试错，逐渐总结出最适合自己的做法。它会从你以前做的菜中挑出最重要的经验，把这些经验压缩成一个秘密包，然后用这个包指导你下一次做菜。这样，你不用每次都依赖老师的指示，而是自己学会了做菜的诀窍。这个过程不断优化，你会变得越来越厉害，最后成为厨房里的大厨，能自己总结经验，做出更好吃的菜。
ELI14 Explained like you're 14
想象你在玩一款游戏。每次玩完后，你会记住一些技巧，比如怎么躲避敌人、怎么得高分。传统上，游戏指南会告诉你一些固定的技巧，然后你照着学。而LOPD就像是你自己在游戏中不断总结经验，把每次成功的招数压缩成一个秘密宝盒，然后用这个宝盒帮助你下一次玩得更厉害。它不会依赖别人给的固定攻略，而是自己从每次游戏中学会最重要的招数。这样，你就能变得越来越厉害，最终成为游戏高手。
Glossary
Self-Distillation (自我蒸馏)
一种模型通过自身轨迹学习，利用内部生成的监督信号提升性能。技术上通过模型自身的输出作为教师，优化自身参数。
本文提出的LOPD就是一种自我蒸馏方法，利用潜在上下文增强监督信号。
Latent Tokens (潜在标记)
连续的、可优化的表示，用于压缩和组织经验信息，作为模型条件输入。技术上通过编码器和压缩器生成。
LOPD中，用潜在标记代替手工设计的特权上下文。
逆KL蒸馏 (Reverse KL)
一种蒸馏损失，鼓励学生模型模仿教师的高概率输出，减少偏差。技术上通过最小化学生分布与教师分布的逆KL散度实现。
用于训练学生模型，使其贴近教师的密集监督。
Privileged Margin (特权边界)
一种约束，确保教师在轨迹成功时具有优势，避免模型崩溃。通过奖励信号调节教师的优势程度。
保证教师提供有价值的监督信号。
Experience Retrieval (经验检索)
从存储的成功轨迹中找到相关经验，为潜在上下文提供内容基础。技术上采用余弦相似度。
为潜在上下文提供丰富的经验信息。
Open Questions Unanswered questions from this research
- 1 如何进一步提升潜在上下文的表达能力，特别是在多模态和长序列任务中？
- 2 如何降低训练复杂度和成本，使LOPD更易于大规模部署？
- 3 未来能否结合元学习实现更强的自适应能力？
Applications
Immediate Applications
自主机器人学习
机器人通过不断尝试和总结经验，自动优化操作策略，无需手工设计规则，提升自主性和适应性。
智能助理个性化优化
利用LOPD实现个性化对话模型的持续学习，自动提取用户偏好和行为模式，增强交互体验。
Long-term Vision
自我演化AI系统
未来AI能自主学习新任务，自动组织经验，持续优化性能，减少人工干预，实现真正的自主学习系统。
Abstract
Enabling agents to learn from experience and internalize it into their policy has become a central problem in self-evolving AI. On-policy self-distillation (OPSD) offers an effective pathway by using a privileged self-teacher to provide dense supervision on the student's own trajectories; however, existing methods still rely heavily on designer-specified privileged artifacts (e.g., answers, feedback, skills, or trajectories), limiting the end-to-end learnability and scalability required for continual self-improvement. In this work, we introduce Latent On-Policy Self-Distillation (LOPD), which, rather than proposing another hand-crafted OPSD variant with a newly prescribed form of privileged context, makes the teacher's privileged context itself learnable end-to-end from experience. Technically, LOPD retrieves relevant experiences and composes them into continuous latent tokens that condition a self-teacher, while the student generates trajectories from the task and interaction history and receives dense token-level supervision at every visited prefix. We further introduce a privileged-margin objective to stabilize and regulate the learning of latent context. Empirically, LOPD demonstrates (I) strong performance, outperforming RLVR and representative OPSD methods including OPSD, SDPO, and Skill-SD across both agentic tool use and code generation; and (II) high learning efficiency, surpassing GRPO and Skill-SD with less than 30% of their rollout budget. Ablation studies further provide direct evidence that making privileged context learnable is necessary for realizing these gains. Together, these results position LOPD as a step toward a more scalable and self-directed paradigm for agent evolution.