Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation

Recent advancements in large-scale pre-trained models have enabled autonomous agents to perform complex tasks through reinforcement learning. Early methods like RLHF and GRPO provided significant progress but struggled with sparse rewards and bias issues, especially when future environment observations were used as privileged information. Self-distillation techniques emerged to refine token-level supervision, yet they often conflated environment feedback with scaffold-induced biases. Addressing these biases is crucial for improving model robustness and generalization, particularly in multi-turn, multi-modal scenarios where environment signals are noisy or incomplete.

The core challenge lies in accurately attributing support signals in privileged replay views, especially when future observations influence token scores. Traditional methods like OPSD re-score tokens under privileged views but cannot disentangle environment feedback from scaffold effects, leading to biased support that hampers learning. This bias becomes more pronounced in high-uncertainty steps, where the model's confidence is low, and the support signals are more susceptible to scaffold-induced shifts. Consequently, models may learn misleading cues, reducing task success and robustness.

The main innovation is the dual-view contrast mechanism that compares a full replay view with an observation-ablated view, isolating the true environment feedback signal. This residual is then used to calibrate token-level updates selectively during uncertain steps, maintaining the overall trajectory direction. Unlike prior work that treats support signals as monolithic, OCSD explicitly separates scaffold effects from environment feedback, providing a more precise supervision signal. The approach integrates step-level selection, sign-preserving mechanisms, and residual modulation, forming a comprehensive bias mitigation framework.

• �� Construct two structurally matched replay views: Full (with future observation) and Observation-Ablated (replacing observation with 'Observation: not provided'); • Compute token support scores in both views, then derive their difference as the observation residual; • Map residuals to [-1,1] range using tanh for calibration; • Select top 20% high-uncertainty steps based on token NLL scores; • Apply residuals to modulate token support in these steps, preserving the sign of the trajectory advantage; • Combine the calibrated token support with the trajectory advantage in the policy update, ensuring stable and bias-reduced learning.

The experimental setup involves training Qwen3 models (1.7B, 4B, 8B) on ALFWorld, WebShop, and Search-QA datasets. The evaluation metrics include success rate, Exact Match (EM), and AUROC for bias detection. The step selection ratio (ρ=0.2) prioritizes uncertain steps for calibration. Baselines include GRPO, OPSD, and SDAR. Multiple ablation studies test the impact of view matching, step selection, and sign alignment. Results show that OCSD consistently outperforms baselines, with success rate improvements of 8-14%, and AUROC increases indicating better bias control.

OCSD achieves success rates of 82.8% on ALFWorld, 73.7% on WebShop, and EM scores of 43.1% on Search-QA, surpassing GRPO by significant margins. The AUROC for bias detection improves from 0.654 to 0.707, confirming better environmental feedback alignment. Ablation results highlight the importance of view matching and step-level calibration, with performance dropping when these components are removed. The residuals show stronger correlation with environment signals, validating the core hypothesis.

This approach is applicable in training autonomous agents for robotics, dialogue, and interactive systems where environment feedback is crucial. It enhances the reliability and robustness of models in dynamic and noisy environments by reducing bias from privileged future observations. The method can be integrated into existing RL frameworks, improving task success and generalization in real-world applications such as service robots, virtual assistants, and adaptive control systems.

The effectiveness depends on accurate environment feedback and view matching, which may be challenging in unstructured or highly noisy settings. Computational overhead, though minimal, could increase with larger models or longer sequences. The current design assumes structured feedback and may require adaptation for multimodal or less predictable environments. Future research should focus on robustness, scalability, and extending applicability to broader scenarios.