Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL
ActObs method supervises observations to change exploration in RL, enhancing Qwen3 model performance on Terminal-Bench 2.0.
Key Findings
Methodology
The ActObs method supervises both actions and observations during SFT, altering RL initialization. It changes the loss mask without adding data or parameters, preserving environment prediction capabilities.
Key Results
- On Qwen3-4B, ActObs achieved 3.4 percentage points higher pass@16 than ActionSFT on Terminal-Bench 2.0.
- In cross-domain aider-polyglot tasks, ActObs outperformed ActionSFT by 4.2 percentage points at pass@1.
- ActObs retained more entropy during RL, requiring less policy movement.
Significance
This research introduces observation supervision to improve RL policy initialization, enhancing task-solving capabilities. The method shows strong cross-domain performance, indicating potential for multi-domain applications.
Technical Contribution
ActObs avoids one-sided gradient specialization by jointly supervising actions and observations, preserving environment prediction and enhancing RL exploration efficiency.
Novelty
This is the first approach to supervise both actions and observations during SFT, altering RL initialization strategies and improving task-solving capabilities.
Limitations
- ActObs shows slightly reduced reliability on single attempts.
- ActionSFT performs better in some cases with larger sampling budgets.
Future Work
Future research could explore applying ActObs in more domains and study its performance across different tasks and model scales.
AI Executive Summary
Policy initialization in reinforcement learning (RL) typically ignores environmental observations, focusing solely on actions. The ActObs method proposed in this paper changes this convention by supervising both actions and observations during the SFT phase. Experiments demonstrate that ActObs significantly enhances task-solving capabilities on Qwen3 models, particularly in Terminal-Bench 2.0 and cross-domain aider-polyglot tasks. By retaining more policy entropy and requiring less policy movement, ActObs improves exploration efficiency. Despite a slight reduction in single-attempt reliability, ActObs excels with larger sampling budgets. Future research could explore its potential in more domains.
Deep Analysis
Background
Reinforcement Learning (RL) often overlooks environmental observations, focusing only on actions. This approach can lead to one-sided policy specialization, limiting exploration efficiency. Recent efforts have aimed to improve RL performance through different initialization strategies, but challenges remain in effectively utilizing environmental observations.
Core Problem
Traditional RL policy initialization ignores environmental observations, leading to a lack of predictive capability regarding environmental changes during exploration. This issue is particularly pronounced in complex tasks, limiting RL's task-solving capabilities.
Innovation
The ActObs method changes RL initialization strategies by supervising both actions and observations during the SFT phase. It modifies the loss mask without adding data or parameters, preserving environment prediction capabilities and enhancing task-solving abilities.
Methodology
- �� During SFT, ActObs supervises both actions and observations.
- �� Changes the loss mask to preserve environment prediction capabilities.
- �� In RL, ActObs retains more policy entropy, requiring less policy movement.
Experiments
Experiments were conducted on Qwen3-4B and Qwen3-8B models using Terminal-Bench 2.0 and aider-polyglot datasets. The effectiveness of ActObs was validated by comparing it with ActionSFT and Obs→Act methods.
Results
ActObs achieved 3.4 percentage points higher pass@16 on Qwen3-4B than ActionSFT and 4.2 percentage points higher pass@1 on aider-polyglot, demonstrating superior performance across tasks.
Applications
ActObs can be applied to RL tasks requiring high exploration efficiency, such as autonomous driving and robotic control. Its cross-domain performance also indicates potential for multi-domain applications.
Limitations & Outlook
ActObs shows slightly reduced reliability on single attempts, and ActionSFT performs better in some cases with larger sampling budgets. Future research should explore how to enhance exploration efficiency without compromising reliability.
Plain Language Accessible to non-experts
Imagine a chef in a kitchen. Traditional RL methods focus only on the chef's actions, ignoring changes in the ingredients. The ActObs method pays attention to both the chef's actions and ingredient changes, helping the chef better predict what to do next. This way, the chef can not only make tastier dishes but also adapt more quickly to new ingredients.
ELI14 Explained like you're 14
Hey there! Imagine you're playing a game with lots of tasks to complete. Traditional methods are like only watching the buttons you press, not what's happening on the screen. ActObs is like paying attention to both the buttons and the screen changes, so you can finish tasks faster! Cool, right?
Glossary
Reinforcement Learning
A machine learning method that trains agents through rewards and penalties.
Used in this paper to train agents for task exploration.
Supervised Learning
A machine learning method that trains models using labeled data.
Used during the SFT phase to train the agent.
Action Supervision
Focuses only on the agent's actions during training.
A strategy in traditional SFT methods.
Observation Supervision
Focuses on both the agent's actions and environmental observations during training.
The core innovation of the ActObs method.
Policy Entropy
Measures the uncertainty of a policy; higher entropy means more diverse strategies.
Used in ActObs to enhance exploration efficiency.
Open Questions Unanswered questions from this research
- 1 How to enhance exploration efficiency without affecting single-attempt reliability?
- 2 How does ActObs perform on larger-scale models?
- 3 How to apply ActObs in more domains?
Applications
Immediate Applications
Autonomous Driving
ActObs can enhance exploration efficiency in autonomous driving systems, helping vehicles better adapt to complex road conditions.
Long-term Vision
Multi-domain Applications
The cross-domain performance of ActObs indicates potential for applications in more complex tasks across various fields.
Abstract
Agent trajectories record what an agent does and what happens next. Yet standard supervised fine-tuning (SFT) applies loss only to agent-authored action tokens, using environment observations as context but not as prediction targets. We ask whether this convention provides the best initialization for subsequent reinforcement learning. We introduce ActObs, which also supervises the observation tokens already present in each trajectory. Although deployed agents never generate observations, learning to predict them encourages the policy to model action consequences without adding data, parameters, sequence tokens, or forward passes. The methods perform similarly after SFT but diverge after GRPO. On Qwen3-4B, GRPO from ActObs achieves higher pass@k at every evaluated sampling budget than its action-only counterpart on Terminal-Bench 2.0. On Qwen3-8B, it trades some pass@1 reliability for higher pass@k (+3.4 pp at pass@16) and solves more distinct tasks. The advantage extends to cross-domain code editing on aider-polyglot (+4.2 pp at pass@1 at 4B), whose tasks are unseen during SFT and RL. ActObs retains more entropy during RL while requiring less policy movement, leaving the final policy closer to its SFT initialization. Our analysis traces this difference to SFT: action and observation gradients rapidly become orthogonal, while action-only training leaves a large residual observation gradient and degrades environment prediction below the base model. Joint supervision prevents this one-sided specialization, preserving consequence prediction and preparing the policy for downstream exploration.