Self-Policy Distillation via Capability-Selective Subspace Projection
Self-Policy Distillation (SPD) extracts low-rank capability subspaces via gradients, guiding self-generation without external signals, improving performance by 13%.
Key Findings
Methodology
SPD computes gradients on correctness-defining tokens from a calibration set, applies SVD to extract dominant directions, and constructs capability subspaces. During self-generation, projection hooks bias KV activations toward these subspaces, steering the model’s outputs. Fine-tuning on generated data refines the model’s ability. The process involves: 1) gradient collection on targeted tokens, 2) SVD for subspace extraction, 3) insertion of projection hooks during decoding, 4) model fine-tuning with generated data.
Key Results
- Across code, math, and QA tasks, SPD improves performance by up to 13% over state-of-the-art self-distillation, with 16% gains on Qwen2.5-14B. In out-of-domain tests, performance increases by 15%, demonstrating strong generalization. Results show SPD effectively filters noise and enhances target capabilities, outperforming baseline methods like SSD and PSR.
- In multiple benchmarks, SPD consistently outperforms models trained with raw self-generated outputs, especially in cross-domain scenarios. The ability subspace extraction reduces style artifacts and task-irrelevant signals, leading to more robust capability transfer.
- The experiments confirm that capability-focused self-generation guided by gradient-derived subspaces significantly boosts model accuracy and robustness, validating the effectiveness of internal capability steering.
Significance
This work addresses the core challenge of enhancing large language models without relying on costly external signals. By leveraging internal gradients to identify and preserve task-relevant capabilities, SPD offers a scalable, domain-agnostic approach to model improvement. Its ability to filter out stylistic and artifact noise enhances robustness and transferability, opening new avenues for autonomous model fine-tuning and capability control. The method’s simplicity, efficiency, and strong empirical results suggest broad applicability in AI development, especially for frontier models where external supervision is scarce or unreliable.
Technical Contribution
The paper introduces a novel gradient-based low-rank subspace extraction mechanism, combining SVD with capability-aligned loss to identify task-relevant directions. It innovatively employs projection hooks during autoregressive decoding to bias self-generation, without modifying model parameters. The integration of LoRA fine-tuning ensures training efficiency. This approach fundamentally differs from existing external-signal-dependent methods, providing a purely internal, scalable, and interpretable capability steering framework with theoretical and empirical validation.
Novelty
This is the first work to utilize model gradients on correctness-defining tokens for capability subspace extraction, combined with projection hooks for internal steering during self-generation. Unlike prior methods relying on external verifiers or reward models, SPD achieves capability selectivity purely through internal signals, enabling better cross-domain generalization and robustness. Its innovative use of SVD for capability filtering marks a significant advance in self-distillation research.
Limitations
- The method depends on a small calibration set for gradient extraction, which may limit effectiveness in low-data or noisy environments.
- Projection hooks are static during inference, potentially limiting adaptability to dynamic task requirements.
- Computational overhead for gradient collection and SVD in very large models may pose scalability challenges, requiring further optimization.
Future Work
Future directions include developing dynamic subspace updating mechanisms, integrating reinforcement learning for adaptive capability steering, and extending the framework to multi-modal models. Exploring online capability refinement and reducing computational costs will further enhance practical deployment. Additionally, combining SPD with other interpretability techniques could improve understanding and control over model capabilities.
AI Executive Summary
Large language models (LLMs) have revolutionized AI with their remarkable capabilities across diverse tasks. However, fine-tuning these models to improve specific skills remains challenging, especially without relying on costly external signals like verifiers or reward models. Traditional self-distillation approaches often suffer from noise and style artifacts, which dilute the signals for targeted capabilities. To address this, the authors propose Self-Policy Distillation (SPD), a novel framework that internally guides a model’s self-generation toward desired capabilities without external supervision.
SPD operates in two main phases. First, it uses a calibration set to compute gradients on correctness-defining tokens, applying singular value decomposition (SVD) to extract a low-rank subspace that captures the core capability directions. Next, during self-generation, projection hooks bias the model’s key-value activations toward this subspace, steering outputs to emphasize target skills. The generated data, filtered through these capability-focused projections, is then used to fine-tune the original model with standard next-token prediction loss.
Experimental results across code, math, and question-answering tasks demonstrate that SPD consistently outperforms existing self-distillation methods, achieving up to 13% improvement over state-of-the-art approaches and 16% over pre-trained baselines. Notably, SPD exhibits superior out-of-domain generalization, with performance gains up to 15%, confirming its ability to enhance model robustness and transferability. The method’s reliance solely on internal signals makes it highly scalable and broadly applicable, especially for frontier models lacking reliable external labels.
This work marks a significant step toward autonomous, capability-oriented model refinement, offering a scalable, interpretable, and effective solution for improving large language models without external supervision. Its potential impact spans automated model tuning, knowledge transfer, and robustness enhancement, paving the way for more adaptable and capable AI systems.
Deep Analysis
Background
近年来，随着GPT、BERT等大规模预训练模型的崛起，模型能力不断扩展，但同时也带来了微调和能力增强的挑战。模型蒸馏技术通过压缩知识提升效率，早期方法如软目标蒸馏（Hinton et al.）依赖外部教师模型，成本高且泛化有限。自蒸馏（Zhang et al.）试图用模型自身生成数据，但受限于噪声和风格干扰，效果不理想。近年来，研究者开始关注能力选择和能力干扰问题，试图在自生成数据中筛选出任务相关的信号，提升模型泛化能力。
Core Problem
现有自蒸馏方法要么依赖昂贵的外部验证信号，要么直接用全部自生成数据，容易强化模型已有偏差，导致能力提升有限。模型自生成输出中，任务相关能力信号常被风格、格式和模型特定错误所淹没，影响能力迁移和鲁棒性。如何在无外部信号的情况下，有效筛选目标能力，提升模型在不同任务和域中的表现，成为亟待解决的难题。
Innovation
本文的创新点包括：1）利用模型梯度在正确性定义标记上提取低秩子空间，明确能力方向；2）引入投影钩子，将KV激活投影到能力子空间，指导自生成偏向目标能力；3）在微调阶段，利用筛选后的自生成数据进行训练，增强目标能力。与传统依赖外部验证器的方法不同，SPD完全依赖模型内部信息，确保能力选择的普适性和效率。
Methodology
- �� 通过校准集计算模型在正确性定义标记上的梯度，形成梯度矩阵。• 对梯度矩阵进行奇异值分解（SVD），提取主要方向，构建低秩子空间。• 在自生成过程中，将KV激活通过钩子投影到子空间，偏向目标能力。• 生成自数据后，移除钩子，利用标准的下一词预测损失微调模型。• 采用LoRA技术，保证训练效率和参数更新的灵活性。• 通过多任务、多域验证方法，确保方法的有效性和泛化能力。
Experiments
在代码生成（MBPP、CodeAlpaca）、数学推理（GSM8K、SVAMP）和问答（MMLU、BBH）等多个任务上，比较基线模型、纯自训练（PSR）、传统自蒸馏（SSD）和提出的SPD。采用不同模型规模（如Qwen2.5-0.5B到14B）进行验证。指标包括准确率、NLL、答题正确率等。通过在源任务、迁移任务和跨域测试中，评估能力提升效果。超参数包括子空间秩r、校准集大小等，均经过调优。
Results
SPD在五个模型上平均提升13%，在Qwen2.5-14B模型上达16%。在跨域测试中，性能提升达15%，明显优于SSD和PSR。能力子空间提取有效过滤噪声，增强目标能力，显著改善模型在未知领域的表现。实验还显示，子空间质量直接影响能力迁移效果，验证了能力选择的关键作用。
Applications
该方法适用于自动微调、知识迁移、模型鲁棒性提升等场景。无需外部验证器，降低成本，提升效率。可应用于多模态模型、知识增强系统，推动AI自主能力调控。未来可结合动态子空间更新和多任务学习，拓展到更复杂的应用场景。
Limitations & Outlook
依赖少量校准集，可能在样本不足或噪声较大时效果受影响。钩子仅在训练阶段使用，推理时未考虑动态调整。大模型中，子空间提取和投影计算成本较高，需优化算法以适应超大规模模型。未来需解决子空间动态更新和多模态扩展的技术难题。
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂里有许多不同的机器，每台机器都能做不同的任务。有时候，你希望让某台机器专注于某个特定任务，比如装配或包装。传统的方法是给机器装上专门的传感器或外部指导，但这样成本高，还不够灵活。本文的方法就像是让工厂自己找到最擅长某项任务的“秘密路径”，然后用这个路径引导机器自己做事。工厂通过观察机器在不同任务中的表现，找到那些最能代表目标能力的“关键路线”，在机器工作时用特殊的“投影器”引导它们。这样，机器就能更专注于目标任务，效果也更好。这种方法不需要外部干预，只依靠工厂内部的观察和调整，就能让机器变得更聪明、更专注。
ELI14 Explained like you're 14
想象你在学校里，有一台超级智能的学习机。平时，它会自己练习各种题目，但有时候它会偏离目标，比如喜欢用奇怪的方式答题，或者记错答案。为了让它更专注于正确的知识点，你可以用一种特别的方法：先观察它在答题时哪里出错，然后找到那些关键的知识点，把这些知识点“提取出来”。接着，在它自己练习的过程中，用一种“魔法”把这些关键知识点放大，让它更容易记住正确的答案。最后，让它用这些“重点”去重新练习，变得更聪明、更准确。这就像是你帮学习机找到最重要的知识线索，让它自己变得更厉害，而不需要外面的老师一直指导。这样，它就能在不同的考试和新题目中表现得更好啦！
Abstract
Self-distillation bootstraps large language models (LLMs) by training on their own generations. However, existing methods either rely on external signals to curate self-generated outputs (e.g., correctness filtering, execution feedback, and reward search), which are costly and unavailable for the best-performing frontier models, or skip curation entirely and train on all raw outputs, an approach that is often domain-specific and hard to generalize. Both also share a deeper weakness that self-generated outputs entangle task-relevant capability with others, such as stylistic patterns, formatting artifacts, and model-specific errors, diluting the signal for the specific capability one aims to improve. In this paper, we propose Self-Policy Distillation (SPD), which achieves generalizable, capability selective without any external signal. Specifically, SPD extracts a low-rank capability subspace from the model's own gradients on correctness-defining tokens, projects key-value (KV) activations into this subspace during self-generation, and fine-tunes on the resulting raw outputs with standard next-token prediction loss. Through extensive experiments across code generation, mathematical reasoning, and multiple-choice QA, we show that SPD achieves up to 13% improvement over state-of-the-art self-distillation methods without external signals and up to 16% improvement over pre-trained baselines. Notably, SPD demonstrates superior generalizability, achieving 15% better performance under out-of-domain generalization settings.