Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents
Proposes BCSD, a dual-view self-distillation framework, improving external skill utilization in LLMs; achieves state-of-the-art results on ALFWorld and WebShop.
Key Findings
Methodology
BCSD integrates self-distillation with reinforcement learning, employing two complementary context views: Meta-Skill for high-level guidance, and pruning view for task-specific focus. Token-level probability gaps (∆aug and ∆pru) are computed between views, used to dynamically rescale advantages (wi,t). The approach enhances skill utilization by combining these signals within the GRPO framework, guiding policy updates more reliably. Meta-Skill is periodically refreshed based on trajectory evidence, ensuring alignment with current policy behavior.
Key Results
- Across multiple model scales, BCSD outperforms baselines; for Qwen2.5-7B, success rate reaches 83.6%, surpassing the best baseline by 3.1 points; WebShop success rate hits 78.1%, 2.4 points above baseline. In Qwen2.5-3B, success exceeds top baselines by 5.4 and 9.4 points respectively.
- Ablation studies reveal Meta-Skill's critical role, with performance dropping significantly when removed. Proper tuning of advantage rescaling coefficient λ (optimal at 0.1) further boosts training stability and effectiveness.
- Multi-view signal fusion improves robustness and generalization, outperforming single-view methods in long-horizon tasks, validating the dual-view self-distillation strategy.
Significance
This work advances reinforcement learning by enabling more effective external skill utilization in LLMs, addressing the bottleneck of sparse rewards and unreliable supervision. It broadens the scope of skill transfer, making models more adaptable and interpretable in complex, multi-task environments. The dual-view mechanism offers a new paradigm for stable, fine-grained policy training, with promising applications in robotics, virtual assistants, and multi-modal systems. The approach paves the way for more autonomous, skill-aware AI agents capable of continual learning and skill refinement.
Technical Contribution
The paper introduces a novel dual-view self-distillation framework that combines high-level Meta-Skill guidance with a pruning-based focus view, utilizing token-level probability gaps to adaptively rescale advantages. This mechanism enhances the stability and reliability of skill utilization signals during reinforcement learning. The integration within GRPO extends the traditional advantage-based policy optimization, providing a more nuanced, context-aware learning process. The periodic Meta-Skill refresh based on trajectory evidence ensures alignment with evolving policy behaviors, offering a theoretically grounded and practically effective method for skill-aware RL.
Novelty
This is the first work to incorporate bidirectional context views into reinforcement learning for skill utilization, explicitly leveraging Meta-Skill and pruning views to address the unreliability of single-context signals. Unlike prior methods that internalize skills or rely solely on sparse rewards, BCSD maintains explicit external skills and guides their use through multi-view token-level signals, representing a significant innovation in skill-aware RL.
Limitations
- Dependence on pre-trained Meta-Skill extraction limits adaptability to unseen tasks or novel skills; generalization remains a challenge. The computational overhead of dual-view evaluation and frequent Meta-Skill updates also constrains large-scale deployment.
- In environments with extremely sparse rewards or high noise, the advantage rescaling mechanism may become less effective, affecting training stability. Further research is needed to improve robustness and efficiency.
- The method assumes the availability of external skills and Meta-Skill extraction pipelines, which may not be feasible in all real-world scenarios. Future work should explore automatic skill discovery and more scalable algorithms.
Future Work
Future directions include developing automatic Meta-Skill generation methods, integrating multi-modal information for richer guidance, and optimizing the efficiency of dual-view computations. Extending the framework to multi-task, continual learning settings, and applying it to real-world robotics and autonomous systems are promising avenues. Additionally, exploring adaptive advantage rescaling strategies and better Meta-Skill generalization will further enhance the method's robustness and applicability.
AI Executive Summary
In the realm of complex decision-making, large language models (LLMs) have demonstrated remarkable capabilities, yet their effective utilization of external skills remains a challenge. Traditional reinforcement learning approaches often rely on sparse outcome rewards, which provide limited feedback on how well the model leverages external guidance. This bottleneck hampers the development of truly skill-aware AI agents capable of performing multi-step, nuanced tasks.
Addressing this, the paper introduces BCSD (Bidirectional Context Self-Distillation), a novel framework that enhances skill utilization by integrating two complementary context views within the RL training process. One view employs Meta-Skill, a high-level guidance mechanism derived from the evolving behavior of the policy, while the other uses a pruning-based approach to focus on task-specific skills by removing redundant information. These views generate token-level probability gaps, which are then combined to dynamically rescale the advantage function during policy optimization.
The core innovation lies in the bidirectional evaluation of trajectories, where the model assesses each response token under both contexts, ensuring more reliable and stable supervision. This mechanism effectively guides the policy to better exploit external skills, especially in long-horizon tasks. Extensive experiments on ALFWorld and WebShop across multiple model scales demonstrate that BCSD consistently outperforms existing baselines, with success rates surpassing previous methods by significant margins.
The results highlight the importance of high-level guidance and multi-view signals in reinforcement learning, providing a new pathway for developing skill-aware, autonomous AI agents. Despite some limitations related to computational costs and generalization, the framework sets a strong foundation for future research in multi-skill transfer, continual learning, and real-world deployment. Overall, BCSD marks a substantial step forward in making language models more adaptable, interpretable, and effective in complex environments.
Deep Analysis
Background
近年来，随着大规模预训练模型的兴起，LLM在多任务场景中展现出强大能力。技能增强方法主要包括存储与检索（如SkillRL、SkillEvolver）和参数内化（如Skill-SD、SDAR），旨在提升技能的重用性和自主学习能力。然而，实际应用中，模型对外部技能的利用仍受限，尤其在长序列任务中，稀疏奖励难以提供细粒度的技能反馈。自蒸馏技术作为一种细粒度监督手段，虽有潜力，但单一上下文信号存在偏差和不稳定的问题。本文旨在通过多视角机制，解决信号偏差和利用效率不足的问题，推动强化学习在复杂环境中的应用。
Core Problem
核心问题在于，现有强化学习方法难以确保模型有效利用外部提供的自然语言技能。稀疏奖励导致对技能应用的反馈不足，模型难以区分技能的有效性。单一上下文的自蒸馏信号易受偏差影响，训练不稳定，影响策略的学习效果。如何设计一种机制，使模型在长序列、多技能环境中，既能充分利用外部技能，又能保持训练的稳定性，成为亟待解决的难题。这关系到智能系统的自主学习和迁移能力，具有重要意义。
Innovation
本研究的创新点在于引入双向上下文视角，结合Meta-Skill和剪枝视角，设计BCSD框架。Meta-Skill提供高层次指导，帮助模型理解技能的正确应用方式；剪枝视角过滤冗余信息，突出任务相关技能。这两视角通过计算token级概率差异，调节优势值，实现优势的动态重调节，增强技能利用效果。与传统单视角方法不同，BCSD实现优势的自适应调节，提升训练稳定性和泛化能力，为强化学习中的技能利用提供新思路。
Methodology
- �� 设计两个互补的上下文视角：Meta-Skill提供高层次指导，剪枝视角过滤冗余信息。
- �� 在训练中，模型在两个视角下评估策略，计算每个token的概率差异∆aug和∆pru，反映不同视角对策略的支持变化。
- �� 利用差异值调节优势值wi,t，通过公式将两个视角的差异融合，动态调节优势。
- �� 在基础的GRPO优化目标中，将调节后的优势值引入，进行策略梯度更新。
- �� 定期更新Meta-Skill，确保其反映模型最新的技能利用状态，避免过时。
- �� 通过多次调参和验证，确保方法的鲁棒性和效果。
Experiments
在ALFWorld和WebShop两个任务上，采用不同规模的Qwen模型（如Qwen2.5-7B、Qwen2.5-3B、Qwen3-1.7B）进行训练。比较基线包括Vanilla、Skill_Prompt、Skill_GRPO、Skill-SD、RLSD、SDAR等。指标主要为成功率和任务得分，训练150步，评估每15步一次。通过消融实验验证Meta-Skill和剪枝视角的贡献，调节优势调度系数λ的影响，以及多视角信号融合的效果。分析训练动态、信号一致性和技能利用情况，确保方法全面验证。
Results
BCSD在所有模型规模中均优于基线，Qwen2.5-7B模型成功率达83.6%，比最优基线高3.1个百分点；WebShop成功率达78.1%，超越最优基线2.4点。Meta-Skill视角的去除导致性能大幅下降，验证其关键作用。调节λ的最优值为0.1，显著提升训练效果。多视角信号融合增强模型稳定性，长序列任务中表现优越，验证了双向自蒸馏的有效性。整体表现证明该方法在复杂任务中显著提升技能利用效率。
Applications
该技术适用于智能助理、机器人自主学习、多任务系统等场景，特别在需要复杂技能调度和迁移的环境中具有潜力。模型在训练中保持外部技能的显式性，便于后续编辑和调优，适合工业界多场景应用。未来结合多模态信息和自动Meta-Skill生成，有望实现更高效的技能迁移和自主学习能力，推动智能系统的广泛应用。
Limitations & Outlook
目前方法依赖预训练Meta-Skill，泛化到新任务或未覆盖技能场景存在挑战。计算成本较高，双视角信号的实时评估增加训练复杂度。在极端稀疏奖励或噪声环境中，优势调节机制可能失效，影响策略稳定性。未来需优化算法效率，增强模型泛化能力，解决实际部署中的成本与鲁棒性问题。
Plain Language Accessible to non-experts
想象你在厨房做饭，手里有一本食谱（技能），但每次做菜时，你都可以根据实际情况调整，比如多放点盐或者少放点油。传统方法就像是把食谱写在纸上，做完就不变了，不能随时改。现在，BCSD就像是有两个助手：一个告诉你怎样更好用食谱（Meta-Skill），另一个帮你过滤掉不重要的步骤（剪枝视角），让你做菜更快更好。两个助手一起帮你判断每一步是不是正确，最后你做的菜味道更棒，也更符合你的口味。这种方法让你在厨房里变得更灵活，能做出更多美味的菜肴。
ELI14 Explained like you're 14
想象你在学校里学做手工艺品，你有一本教程（技能），但每次做都不一样。有时候你会忘记一些步骤，或者觉得某些部分不重要。传统的方法就是死记硬背教程，做完就完事。现在，BCSD就像是有两个朋友在帮你：一个告诉你怎么用教程（Meta-Skill），让你知道哪些步骤最重要；另一个帮你删掉不必要的部分（剪枝视角），让你专注在关键步骤上。两个朋友会不断告诉你每一步是不是正确，帮助你做出更漂亮的作品。这样，你就能更灵活、更聪明地完成手工艺品，也能学到更多技能。
Glossary
Meta-Skill
一种高层次的技能指导，帮助模型理解如何有效应用外部技能。技术上是模型在训练中学习到的关于技能使用的策略。
在论文中，Meta-Skill用作指导模型在不同任务中正确利用技能的高层次信息。
Self-Distillation
模型利用自身预测作为教师，进行细粒度监督，从而提升性能。技术上是模型在不同视角或状态下相互指导。
论文中采用双向自蒸馏机制，增强模型对技能的利用能力。
Advantage
强化学习中，用于衡量某行动相对于平均水平的优劣。公式为A=Q−V。
在本文中，优势值经过调节，用于引导策略优化。
GRPO
一种基于群体奖励的策略优化算法，考虑多个样本的奖励均值和标准差。公式为A_GRPO=（r_i−r̄）/σ_r。
作为基础优化目标，结合优势调节机制提升技能利用。
Token
自然语言处理中的最小单位，可以是字、词或子词。模型通过预测词元生成文本。
论文中的token级信号用于细粒度调节策略。
Open Questions Unanswered questions from this research
- 1 如何自动生成更具泛化能力的Meta-Skill，以适应未见任务或新技能场景，仍是未解难题。现有方法多依赖预训练，泛化能力有限。
- 2 多视角信号融合的理论基础尚不完善，如何在不同任务中动态调节视角权重，提升鲁棒性，是未来研究方向。
- 3 在极端稀疏奖励或高噪声环境下，优势调节机制的稳定性和效果仍需验证，需开发更鲁棒的调节策略。
Applications
Immediate Applications
Skill Scheduling in Virtual Assistants
Enhances multi-task handling by better leveraging external skills, improving user interaction quality. Maintains explicit skills for easy editing and refinement.
Autonomous Robot Learning
Allows robots to better understand and apply external instructions in complex environments, improving task efficiency and transferability.
Long-term Vision
Multi-Skill, Multi-Task AI Systems
Develop autonomous agents capable of continual skill acquisition and transfer across diverse domains, revolutionizing automation and intelligent systems.
Abstract
External natural-language skills provide large language model (LLM) agents with reusable and editable guidance for solving complex tasks. Yet their effectiveness depends not only on skill quality, but also on whether the policy can translate the provided guidance into appropriate actions. However, methods specifically designed to improve this skill-utilization ability remain largely underexplored. In practice, skill-based agents are commonly trained with reinforcement learning objectives centered on task-level rewards, which offer limited supervision and struggle to capture subtle differences in how effectively the policy uses the provided skills. We propose BCSD (Bidirectional Context Self-Distillation), a framework that combines self-distillation with reinforcement learning to train LLM agents to use external skills more effectively. Unlike prior self-distillation methods that rely on a single privileged context, BCSD evaluates each trajectory from two complementary skill-context views. The augmented view introduces higher-level Meta-Skill guidance, while the reduced view prunes general guidance to highlight task-specific skills. Their complementary token-level signals are combined to rescale the RL advantage. Experiments on ALFWorld and WebShop demonstrate that BCSD achieves the strongest overall performance across model scales, enabling agents to utilize external skills more effectively. Ablation studies further verify the complementary contributions of the augmented and reduced context views. Code will be released to ensure full reproducibility.