AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning
AgentOPSD employs recursive Bayesian belief updates for turn-level credit assignment, outperforming baselines with 89.1% success on ALFWorld.
Key Findings
Methodology
AgentOPSD introduces token-level teacher-student log-probability gaps aggregated into turn-level evidence, recursively updating a Bayesian belief state in log-odds space. This process transforms sparse outcome supervision into dense turn-level credit signals, identifying pivotal turns via marginal belief revisions. The approach leverages environment success signals without requiring critics or extra rollouts, aligning token-level signals with environment transitions. The core components include token gap aggregation, recursive belief updates, and outcome-aligned credit normalization, enabling precise credit assignment in long-horizon interactive tasks.
Key Results
- On ALFWorld with Qwen2.5-7B, AgentOPSD achieved 89.1% success, surpassing GRPO and other self-distillation baselines. It demonstrated robustness against increasing task horizon, with less performance degradation (−0.54 success points per turn) compared to −2.91 (GRPO) and −3.59 (RLSD). Ablation studies confirmed turn-level aggregation and recursive belief updates as critical for performance gains.
- Across WebShop and Search-QA, AgentOPSD maintained superior success rates, especially on longer tasks. It consistently outperformed baselines in both 3B and 7B models, with significant improvements in success stability as task length increased, validating its effectiveness in long-horizon settings.
- Ablation experiments showed that removing turn boundary aggregation or replacing recursive belief updates with local gaps reduced success by at least 4-6 percentage points, emphasizing the importance of the belief revision framework for accurate credit assignment.
Significance
This work advances reinforcement learning by providing a theoretically grounded, critic-free method for turn-level credit assignment in long-horizon tasks. By integrating token-level self-distillation signals into a recursive Bayesian framework, it addresses the core challenge of sparse reward crediting, enabling more efficient and robust learning in complex interactive environments. The approach has broad implications for autonomous agents in robotics, dialogue systems, and decision-making tasks, where understanding the influence of individual actions over extended sequences is crucial.
Technical Contribution
The key technical innovation lies in formalizing turn credit as a belief revision process in log-odds space, driven by token-level self-distillation gaps. This enables recursive, history-dependent credit assignment without critic networks or additional rollouts. The method combines environment success signals with token-level evidence, dynamically updating a belief state that highlights pivotal decisions. It also introduces a bounded advantage reshaping scheme, stabilizing credit signals across trajectories, and demonstrates compatibility with standard policy optimization algorithms, significantly improving long-horizon task performance.
Novelty
This research is the first to systematically leverage token-level self-distillation differences as evidence for turn-level credit via recursive Bayesian belief updates. Unlike prior methods that broadcast a global advantage uniformly, AgentOPSD dynamically adjusts credit based on history-dependent belief revisions, providing a more nuanced and effective approach for long-horizon reinforcement learning. Its critic-free design and environment-aligned evidence aggregation mark a significant departure from existing techniques.
Limitations
- The method relies on verifiable, well-defined terminal rewards; in environments with noisy or sparse signals, belief updates may become unreliable.
- In extremely long sequences or high-noise scenarios, belief recursion might accumulate errors, impacting pivotal turn detection.
- Computational overhead increases with sequence length due to recursive belief updates and token-level aggregation, potentially limiting scalability.
Future Work
Future directions include integrating learned models for belief calibration, extending to multi-modal environments, and optimizing computational efficiency. Exploring adaptive decay factors and more sophisticated evidence aggregation could further enhance robustness. Additionally, applying this framework to multi-agent systems and real-world robotics tasks presents promising avenues for research.
AI Executive Summary
Long-horizon interactive tasks in reinforcement learning often suffer from sparse reward signals, making effective credit assignment a persistent challenge. Traditional approaches like GRPO broadcast a uniform advantage across entire trajectories, which diminishes in effectiveness as task length increases. Recognizing this limitation, the authors propose AgentOPSD, a novel method that leverages token-level self-distillation signals to infer turn-level contributions. The core idea is to interpret token-level log-probability gaps as evidence that can update a success belief recursively in log-odds space, aligning credit with pivotal decisions. This belief update process is inspired by Bayesian principles, enabling the model to dynamically identify critical turns that influence the final outcome without requiring critics or additional environment rollouts. The approach is fully compatible with standard policy optimization algorithms, making it practical for large-scale deployment.
Experimental results across three diverse environments—ALFWorld, WebShop, and Search-QA—demonstrate the effectiveness of AgentOPSD. In particular, on ALFWorld with Qwen2.5-7B, success rates reach 89.1%, outperforming strong baselines such as GRPO and self-distillation variants. The method shows remarkable robustness as task horizons extend, with significantly less performance degradation compared to traditional methods. Ablation studies further confirm that the key components—turn boundary aggregation and recursive belief revision—are essential for this success. The findings suggest that recursive, history-dependent credit assignment offers a promising pathway for improving long-horizon reinforcement learning, with broad implications for autonomous systems, dialogue agents, and complex decision-making tasks. Despite its strengths, the approach faces challenges in environments with noisy rewards or extremely long sequences, where belief recursion may accumulate errors. Future work will focus on enhancing robustness, scalability, and multi-modal extensions, aiming to bring this innovative credit assignment framework closer to real-world applications.
Deep Analysis
Background
强化学习在自主智能体中的应用不断深化，早期多采用奖励驱动策略（Sutton & Barto, 2018），但面对长序列任务，奖励稀疏成为瓶颈。近年来，优势函数方法（如TRPO、PPO）提升了样本效率，但在长时序中，信用分配仍难以精确。GRPO等策略通过全局优势广播改善表现，但无法区分关键决策。token级监督（Ye et al., 2026a）提供更密集信号，但难以结合环境交互。本文提出递归贝叶斯信念机制，旨在实现更细粒度的回合信用分配，解决稀疏奖励问题。
Core Problem
长序列交互环境中，稀疏奖励限制了学习效率。现有方法难以识别哪些决策对最终结果影响最大，导致训练信号分散，影响策略优化。尤其在多轮任务中，单一奖励难以反映中间关键步骤，造成误导。此问题在自动问答、机器人导航等应用中尤为突出，亟需一种动态识别关键决策的机制，以提升学习效果。
Innovation
本研究的创新点包括：1）引入贝叶斯信念递归机制，将token级差异转化为回合级信号，增强历史依赖性；2）设计无critic的端到端信用分配方案，简化模型结构；3）结合环境终局奖励，动态调节关键回合的信用权重，提升长序列表现。这些创新突破了传统全局优势广播的局限，为长时序强化学习提供了新思路。
Methodology
- �� 任务定义：在多轮交互中，智能体基于观察选择动作，环境反馈终局奖励。• Token差异：利用教师-学生模型，计算每个token的对数概率差异，作为局部证据。• 聚合为回合证据：将token差异求和，得到回合证据。• 贝叶斯信念递归：在对数几率空间中，初始化成功信念B0，从每个回合的证据更新信念状态Bk（公式8）。• 关键回合识别：通过边际修正∆Bk，衡量每个回合对成功概率的影响。• 信念修正：结合奖励，调节信念变化，识别关键决策。• 信用重塑：标准化和边界限制信用值，确保稳定。• 训练目标：最大化信念修正带来的奖励提升。
Experiments
在ALFWorld、WebShop、Search-QA环境中，比较AgentOPSD与GRPO、SDAR等基线。模型规模为3B和7B。指标包括成功率、准确率和奖励。超参数如λ（信念重塑）、γ（证据衰减）和ε（策略剪切）调节模型。通过消融验证机制贡献，特别是边界对齐和信念递归。在长序列任务中，AgentOPSD表现优异，成功率提升至少10%。
Results
在ALFWorld中，Qwen2.5-7B模型成功率由91.2%提升至89.1%，表现出更强鲁棒性。WebShop和Search-QA中，成功率在长任务中保持优越，误差最小。消融显示，边界聚合和信念递归是性能提升的关键，去除任何一环都显著降低效果。整体验证了其在长序列强化学习中的优势。
Applications
适用于机器人导航、智能问答、自动客服等长序列交互场景。通过识别关键决策，提高系统效率和鲁棒性。未来可结合多模态信息，扩展到视觉、语音等多感知环境，推动自主系统智能化。
Limitations & Outlook
依赖环境奖励的真实性，奖励稀疏或不准时，信念更新会偏离实际。在极长序列或噪声环境中，信念递归可能积累误差，增加计算负担。未来需优化算法，增强鲁棒性和效率，同时考虑多模态融合。
Plain Language Accessible to non-experts
想象你在一家工厂工作，每天都要完成一系列任务，比如组装、检查、包装。每个环节都很重要，但你不知道哪些步骤最关键，因为你只知道最后的成品是否合格。现在，工厂引入了一种新工具，能根据每个步骤的表现，判断哪些操作对最终结果影响最大。这个工具会不断学习，记住哪些操作是成功的关键，哪些是多余的。这样，你就能专注于那些真正重要的步骤，提高效率。这个工具就像论文中的AgentOPSD，通过不断分析每个决策的影响，帮助智能体在复杂任务中找到最关键的转折点，提升整体表现。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，每一轮你都要做很多决定，但你只知道最后是否赢了。你可能不知道哪些决定最重要，但你希望找到那些真正让你赢的关键时刻。这篇论文介绍了一种聪明的方法，能帮你在游戏中找到这些关键时刻。它会观察每个决定的细节，然后用一种特别的数学方法，逐步判断哪些决定改变了结果。就像你在游戏中学会了哪些动作最有效，下一次你就能更聪明地做决定。这个方法不用额外的练习，也不需要反复试错，就能帮你变得更厉害，特别是在那些需要很多回合才能知道结果的复杂任务中。
Abstract
Reinforcement learning (RL) with verifiable rewards constructs trajectory-level advantage estimates, yet it often fails to credit the few pivotal decisions that determine outcomes in long-horizon, multi-turn agentic tasks. Recent work introduces privileged self-distillation for credit assignment, providing denser supervision, but it remains unclear how such local signals should represent sequential credit. We propose AgentOPSD, a critic-free, recursive method for turn-level credit assignment in agentic reinforcement learning. AgentOPSD aggregates token-level teacher-student log-probability gaps into turn-level evidence and recursively updates a Bayesian belief state in log-odds space. This yields a principled reweighting scheme that converts sparse outcome supervision into turn-level credit signals and identifies pivotal turns through the marginal belief revision between consecutive states. The method is fully compatible with standard policy optimization and requires neither an additional critic nor extra rollouts. We evaluate AgentOPSD on ALFWorld, WebShop, and Search-QA using Qwen2.5 models at two scales (3B and 7B). AgentOPSD outperforms GRPO and strong self-distillation baselines, achieving 89.1% success on ALFWorld with Qwen2.5-7B. Ablation studies attribute the gains to turn-level aggregation and history-dependent recursive belief updates.