Trust Is Not Enough: Influence Calibration for On-Policy Self-Distillation in Agentic RL
Proposes Influence Calibration (ICSD) to address trust-utility mismatch, improving task success rates by 3-5% on benchmarks.
Key Findings
Methodology
This paper introduces Influence Calibration for Self-Distillation (ICSD), which quantifies each token’s impact on the current RL objective by measuring the first-order response of its importance-weighted surrogate to teacher-directed output perturbations. Batch-adaptive calibration transforms these non-stationary influence scores into bounded weights, ensuring stable distribution across training stages. ICSD leverages existing policy gradients, avoiding additional forward passes, and combines influence scores with trust signals to reallocate auxiliary supervision. The approach maintains total support mass via exact action-turn mass matching, leading to consistent improvements across multiple tasks and models.
Key Results
- In ALFWorld, ICSD reduces teacher-supported objective-opposed tokens from 60.1% to 37.8%, and achieves 96.1% success rate with 7B models, outperforming trust-only methods. In WebShop, scores increase to 93.1, with similar gains across other benchmarks.
- Across three benchmarks and multiple model sizes (1.5B-7B), ICSD consistently outperforms trust-based allocation, with average improvements exceeding 3 percentage points in success and accuracy metrics.
- Ablation studies show that combining influence scores with trust signals yields the best performance, with influence alone raising compatibility with RL gradients by 0.192, confirming the importance of objective-aware supervision.
Significance
This work addresses a fundamental challenge in reinforcement learning: aligning supervision signals with current objectives. By quantifying the actual influence of teacher guidance on policy improvement, ICSD enhances sample efficiency, robustness, and generalization in multi-task language agents. The method bridges the gap between trustworthiness and utility, enabling more effective knowledge reuse and policy refinement, with broad implications for autonomous systems and AI safety.
Technical Contribution
The paper introduces a novel influence score derived from teacher perturbations, capturing the first-order response of the RL surrogate. It integrates this with batch-adaptive calibration, employing asymmetric Laplace functions to stabilize scores across training stages. The exact action-turn mass matching ensures total supervision mass remains constant, providing stability and interpretability. These innovations collectively enable objective-informed supervision allocation without additional model passes, advancing the state-of-the-art in on-policy self-distillation.
Novelty
This is the first work to incorporate teacher-directed influence scores based on output perturbations into self-distillation for RL agents. Unlike prior trust-based or uncertainty-based methods, ICSD explicitly measures how each token’s supervision affects the current policy’s improvement, leading to more effective and targeted supervision. The combination of influence measurement, batch calibration, and mass matching constitutes a new paradigm for supervision allocation in reinforcement learning.
Limitations
- The influence estimation relies on linear approximation, which may be less accurate under high noise or distribution shifts. The method’s computational overhead, though minimal, could be significant for very large models or long sequences. Further research is needed to extend influence measures beyond first-order and linear assumptions, especially in highly non-stationary environments.
Future Work
Future directions include extending influence estimation to higher-order and nonlinear effects, integrating adaptive perturbation strategies, and applying the framework to multi-modal and continuous control tasks. Exploring automatic tuning of calibration parameters and scaling to larger models will further enhance robustness. Additionally, combining influence calibration with other supervision signals could unlock new levels of efficiency and generalization in autonomous learning systems.
AI Executive Summary
Reinforcement learning for language agents often suffers from delayed and sparse rewards, limiting effective policy improvement. Recent advances introduced dense auxiliary supervision via self-distillation, where privileged teachers provide token-level guidance. However, traditional methods rely heavily on trust signals—measures of teacher reliability—that do not necessarily reflect whether the supervision supports the current policy’s objectives. This trust-utility mismatch hampers the efficiency and effectiveness of self-distillation, especially in complex, multi-task environments.
To address this, the paper proposes Influence Calibration for Self-Distillation (ICSD), a novel framework that quantifies each token’s actual influence on the current RL objective. By measuring the first-order response of importance-weighted surrogate contributions to teacher-directed output perturbations, ICSD captures the true utility of supervision signals. The influence scores are then stabilized via batch-adaptive calibration, which employs asymmetric Laplace functions to produce bounded, comparable weights across training stages. This process ensures that supervision is reallocated based on objective relevance rather than trust alone.
A key innovation is the exact action-turn mass matching, which preserves the total auxiliary loss support while redistributing it toward tokens that genuinely support policy improvement. Extensive experiments across ALFWorld, WebShop, and Search-QA benchmarks demonstrate that ICSD consistently outperforms trust-only allocation strategies, improving success rates and scores by several percentage points. For instance, in a 7B model, ALFWorld success reaches 96.1%, and WebShop scores hit 93.1, surpassing baseline methods.
Analysis shows that ICSD significantly reduces teacher-supported mass on objective-opposed tokens from 60.1% to 37.8%, and enhances the alignment between the policy gradient and the influence signals. These results highlight the importance of objective-aware supervision, enabling more efficient and stable policy updates. Overall, ICSD provides a principled, scalable solution to improve on-policy self-distillation, with broad implications for reinforcement learning, language modeling, and autonomous systems.
Deep Analysis
Background
强化学习在长序列任务中面临奖励稀疏和延迟的问题，限制了策略的有效优化。近年来，密集局部监督技术如自我蒸馏被引入，显著改善了训练信号的密度。代表性工作如SDAR（Lu et al. 2026）通过利用教师提供的偏好信息，增强模型的学习效率。然而，这些方法多依赖信任估计指标（如不确定性、位置偏差），未能充分考虑支持信号对当前目标的实际贡献，导致信任-效用失配。这在多任务、多模型环境中尤为明显，限制了自我蒸馏的潜力。
Core Problem
核心问题在于如何合理分配教师提供的密集监督信号，使其真正促进当前策略的改进。传统方法依赖信任指标（如不确定性、位置偏差、 divergence），但这些指标不能反映支持信号是否符合当前目标，导致大量目标反向tokens被支持，影响训练效果。信任-效用失配严重制约了自我蒸馏在复杂环境中的应用效果。解决方案需要引入目标影响的量化指标，确保监督信号的实用性和有效性。
Innovation
本文的主要创新包括：1）提出基于教师输出扰动的目标影响指标，量化每个token对目标的实际贡献；2）结合批次自适应校准机制，用非线性函数稳定影响分数，避免极端偏差；3）引入exact action-turn mass matching，确保总支持量不变，提升算法稳定性。这些创新突破了传统信任估计的局限，为策略优化提供了更精细的目标对齐工具，显著提升了训练效率和策略质量。
Methodology
- �� 计算教师引导扰动下的第一阶影响响应，作为目标影响指标；• 采用中位数和偏差尺度对影响分数进行批次校准，限制极端值；• 利用非线性校准函数（如不对称Laplace）将影响分数映射到有界范围；• 结合信任信号，调整每个token的支持比例，确保支持的目标相关性；• 通过exact action-turn mass matching，保证总支持量不变，保持训练稳定性。这一流程利用已有梯度信息，无需额外模型前向，提升了效率。
Experiments
在ALFWorld、WebShop和Search-QA三个任务上，比较ICSD与传统信任基础方法的性能。模型包括1.5B、3B、7B参数的Qwen系列，采用GRPO和GiGPO优化器。指标涵盖成功率、得分和准确率。通过消融实验验证影响指标的贡献，观察目标反向tokens比例变化，分析与RL梯度的相似性。实验设计确保多样性和代表性，验证ICSD的广泛适用性和有效性。
Results
ICSD在所有任务中均优于信任单一策略，ALFWorld成功率提升至96.1%，WebShop得分达93.1，均优于对比方法。消融分析显示，ICSD显著降低目标反向tokens比例，从60.1%降至37.8%，同时提升策略与RL梯度的余弦相似性0.192。多模型、多任务环境中，ICSD表现出强大的泛化能力和稳定性，验证了其在复杂强化学习场景中的潜力。
Applications
该方法适用于需要密集局部监督的长序列任务，如对话系统、机器人控制和复杂策略学习。通过优化目标对齐，提升模型训练效率和鲁棒性。未来可结合多阶影响和非线性校准，扩展到多模态和自主系统中，推动自主学习的智能化发展。
Limitations & Outlook
当前方法依赖线性扰动近似，可能在高噪声或极端分布偏移环境中表现不足。对大规模模型和长序列任务，计算成本仍较高。未来需探索非线性影响建模和多阶校准，以增强鲁棒性和适应性。
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂里有许多工人（模型），他们需要合作完成一项复杂的任务（比如组装一台电脑）。每个工人都听从老师（教师模型）的建议，但老师的建议有时并不完全符合当前的生产目标。以前的方法只看老师建议是否可信（信任），但没有考虑这些建议是否真的帮助工人完成任务（效用）。现在，有个聪明的管理者（算法）会衡量每个建议对当前任务的实际帮助（影响），然后根据这个影响调整建议的分配。这样，工人们就能更专注于真正重要的部分，工作效率更高，产品质量也更好。这就像用“影响校准”让工厂的建议更贴合实际生产需要，从而整体提升效率。
ELI14 Explained like you're 14
想象你在学校做项目，老师会给你很多建议，但有些建议特别重要，有些则不那么关键。以前，你只听老师说“这个建议可信”就采纳，但其实，有些建议虽然可信，但对你完成项目帮助不大。现在，假如你有个聪明的朋友，他会告诉你每个建议到底对你完成任务的帮助有多大，然后帮你决定重点听哪个建议。这样，你就能用有限的时间，专注在最重要的部分，项目做得更棒。这就像论文里的“影响校准”，它帮模型判断每个老师建议对当前目标的实际帮助，从而更聪明地学习，效果更好！
Abstract
On-policy self-distillation (OPSD) gives language agents dense token-level supervision from a privileged self-teacher on the policy's own trajectories. Existing methods allocate this supervision mainly by teacher trust, but trust does not reveal whether emphasizing a token supports the current policy objective. We call this the trust-utility mismatch and introduce Influence Calibration for Self-Distillation (ICSD). For each supervised token, ICSD measures the first-order response of its importance-weighted RL surrogate contribution to a teacher-directed output perturbation. Batch-adaptive calibration converts this non-stationary signal into a bounded allocation weight while preserving the original auxiliary-loss mass within each action turn. These detached weights affect only the distillation loss and require no additional model pass. Across ALFWorld, WebShop, and Search-QA, ICSD improves all matched aggregate metrics over trust-only allocation under Group Relative Policy Optimization (GRPO) and Group-in-Group Policy Optimization (GiGPO), across two model families spanning 1.5B to 7B. At 7B, it reaches 96.1% ALFWorld success and a WebShop score of 93.1. Frozen-batch analyses show that ICSD reduces teacher-supported mass assigned to objective-opposed tokens from 60.1% to 37.8% and raises cosine compatibility with the RL gradient by 0.192. A companion repository is avail- able at https://github.com/lanqz7766/Influence-Calibration-for-On-Policy-Self-Distillation-in-Agentic-RL.