ECHO: Terminal Agents Learn World Models for Free
ECHO integrates environment observation prediction with policy optimization, doubling task success rate from 2.70% to 5.17% on Qwen3-8B in TerminalBench-2.0.
Key Findings
Methodology
This paper introduces ECHO (Environment Cross-entropy Hybrid Objective), which combines standard policy-gradient loss with an auxiliary environment observation prediction loss. By reusing the same forward pass as GRPO, ECHO predicts environment responses (stdout, errors, logs) as dense supervision signals, leveraging the environment stream as on-policy training data. This approach enhances the model’s understanding of environment dynamics without additional rollouts or teacher models, enabling self-evolving curriculum and better generalization, especially on unseen trajectories. Experiments demonstrate significant improvements in task success rates and environment prediction accuracy across multiple models and benchmarks.
Key Results
- ECHO nearly doubles the pass@1 rate on TerminalBench-2.0 from 2.70% to 5.17% for Qwen3-8B, and from 5.17% to 10.79% for Qwen3-14B, outperforming GRPO alone. It reduces environment token cross-entropy sharply, indicating improved environment modeling. Even on off-policy trajectories generated by stronger models, ECHO significantly lowers prediction error, showing transferability of learned environment dynamics.
- Without relying on expert demonstrations, ECHO matches or exceeds the performance of models trained with 15k expert trajectories, recovering about half of the initial advantage. It also enables verifier-free self-improvement, where models improve solely through environment interaction and response prediction.
- Training dynamics reveal faster convergence and higher final performance, with ECHO reaching optimal scores in fewer steps and reducing inference timeouts and token usage, thus improving efficiency and robustness.
Significance
This work fundamentally shifts the paradigm in reinforcement learning for language-based agents by exploiting environment responses as dense, on-policy supervision signals. It addresses the core challenge of reward sparsity, enhances the model’s ability to understand and predict environment behavior, and promotes autonomous self-improvement. The approach has broad implications for deploying intelligent agents in real-world scenarios, where explicit reward signals are scarce or delayed. By enabling models to learn from environment feedback directly, ECHO paves the way for more autonomous, adaptable, and efficient AI systems capable of complex interactions without heavy reliance on external supervision.
Technical Contribution
The key technical innovation is the hybrid loss that integrates environment observation prediction into the policy gradient framework, reusing the same logits for both tasks. This on-policy, dense supervision mechanism improves the model’s internal world model, leading to better environment dynamics prediction and task generalization. The method is compatible with existing stabilization techniques and requires no additional rollouts or teacher models, making it computationally efficient. It demonstrates a new way to incorporate environment feedback into reinforcement learning, bridging the gap between language modeling and world modeling.
Novelty
This is the first work to treat environment responses as dense, on-policy supervision signals within reinforcement learning for language agents. Unlike prior methods relying solely on sparse rewards or external demonstrations, ECHO leverages the environment stream itself as a continuous training target, enabling models to self-supervise their understanding of environment dynamics. This approach introduces a novel, scalable way to improve policy learning and generalization in complex interactive environments.
Limitations
- The effectiveness depends on the structure and clarity of environment responses; ambiguous or noisy outputs may hinder learning.
- Scaling to highly complex, high-dimensional environments remains challenging due to model capacity and training costs.
- While promising, the method’s transfer to real-world robotics or unstructured environments requires further validation and adaptation.
Future Work
Future research will explore integrating multi-modal environmental signals, extending the approach to physical robots, and combining with imitation learning for higher-level strategy acquisition. Additionally, improving robustness in noisy or unstructured environments and reducing computational costs for large-scale deployment are key directions.
AI Executive Summary
This study introduces ECHO, a novel reinforcement learning framework that leverages environment responses as dense supervision signals for terminal agents. Traditional RL approaches in language models often suffer from sparse rewards, limiting learning efficiency and generalization. ECHO addresses this by reusing the model’s own output during rollouts to predict environment observations, such as stdout logs, error messages, and file contents, turning these into dense training targets. This method effectively transforms the environment stream into a continuous, on-policy curriculum, enabling the model to learn environment dynamics directly from interactions.
Experimental results demonstrate that ECHO significantly boosts task success rates across multiple models and benchmarks. For instance, on TerminalBench-2.0, Qwen3-8B’s pass@1 rate nearly doubles from 2.70% to 5.17%, and Qwen3-14B from 5.17% to 10.79%. Notably, ECHO improves environment prediction accuracy, as evidenced by sharply reduced cross-entropy on off-policy trajectories generated by stronger models. These improvements occur without external demonstrations, as ECHO can recover roughly half of the performance gains typically provided by expert data, highlighting its capacity for autonomous self-improvement.
The core innovation lies in integrating environment observation prediction into the policy gradient training, enabling the model to develop a richer internal world model. This approach enhances the agent’s understanding of environment dynamics, leading to better generalization and efficiency. The method is computationally efficient, requiring no additional rollouts or teacher models, and is compatible with existing stabilization techniques.
Overall, ECHO represents a significant step forward in autonomous reinforcement learning, offering a scalable way to harness environment feedback for continuous self-improvement. Its ability to learn environment dynamics directly from interactions opens new avenues for deploying intelligent agents in real-world, complex scenarios, reducing reliance on external supervision and enabling more adaptable AI systems.
Deep Analysis
Background
近年来，强化学习结合自然语言处理的终端代理逐渐成为研究焦点。早期工作如GPT-4在任务执行中依赖稀疏奖励，导致学习效率低下。GRPO（Group-Relative Policy Optimization）提出了基于剪辑优势的策略优化，但仍受限于奖励稀疏问题。随着环境模拟和自我监督技术的发展，研究者开始尝试利用环境观察作为密集信号，提升模型对环境动态的理解。此前多依赖外部示范或奖励信号，缺乏系统性利用环境响应的方案。本文创新性地提出ECHO，将环境观察作为密集监督信号，显著改善学习效率和泛化能力，为自主学习提供新思路。
Core Problem
传统终端代理在复杂环境中面临奖励稀疏、学习缓慢的问题。现有方法多依赖外部示范或奖励信号，难以充分利用环境反馈，导致模型对环境动态理解不足，限制自主学习和泛化。如何利用环境观察作为密集训练信号，提升模型对环境响应的预测能力，成为核心难题。尤其在未见轨迹和新任务中，模型的泛化能力不足，限制了实际应用潜力。
Innovation
本研究的创新在于提出ECHO，将环境观察作为密集训练目标，结合策略梯度优化实现自我监督。具体包括：1）利用模型输出作为观察预测目标，避免额外采样；2）复用前向传播，无需额外计算；3）在多任务环境中验证，显著提升任务成功率和环境动态预测能力。这一方法突破了稀疏奖励限制，提供了新思路，增强模型理解和泛化能力。
Methodology
- �� 采用GRPO作为基础框架，结合环境观察预测作为辅助目标。
- �� 在每次rollout中，模型输出动作和环境响应，利用环境响应作为预测目标，计算交叉熵损失。
- �� 在模型输出的相应位置加入环境观察预测的辅助损失，实现密集监督。
- �� 复用模型前向传播，避免额外计算成本，训练过程中同时优化策略和环境动态理解。
- �� 设置环境观察预测的权重λ，调节策略优化与观察预测的平衡。
- �� 在多任务终端环境中，验证模型在未见轨迹上的泛化能力和任务成功率提升。
Experiments
采用多任务终端环境（如TerminalBench-2.0）进行评估，比较GRPO与ECHO的性能差异。训练数据包括由GPT-5生成的8870个任务样本，模型在不同规模（8B、14B）上训练，设置λ=0.05。指标包括任务成功率（pass@1）、环境token交叉熵等。通过在未见轨迹上的预测能力验证模型对环境动态的理解。还进行了不同模型初始化（如SFT、GRPO）和对比分析，验证ECHO在泛化和训练效率上的优势。
Results
ECHO在多个模型和任务上均表现优异，Qwen3-8B模型的pass@1从2.70%提升到5.17%，Qwen3-14B从5.17%提升到10.79%。环境token的交叉熵显著降低，表明环境动态预测能力增强。未依赖专家示范，模型也能匹配甚至超越基于示范的性能，显示出强大的自主学习能力。训练速度提升，超时率降低，推理效率增强，验证了方法的实用性和有效性。
Applications
该技术适用于自动化系统、机器人自主操作、复杂任务自动化等场景，尤其在缺乏大量标注数据或示范的环境中展现出巨大潜力。通过利用环境响应作为密集信号，减少对外部示范的依赖，提升自主学习和适应能力。未来可结合多模态信息，拓展到真实世界的机器人和交互系统，实现更智能的自主操作。
Limitations & Outlook
目前方法依赖环境响应的结构化输出，复杂或模糊的环境响应可能影响预测效果。模型在极端复杂或高维环境中泛化能力有限，训练成本较高。未来需结合多模态信息和更丰富的环境特征，提升泛化和鲁棒性。
Plain Language Accessible to non-experts
想象你在厨房做饭，菜谱就像模型的任务，厨师（模型）需要根据食材（环境信息）做出决定。传统方法就像只告诉厨师“做菜成功或失败”，但没有告诉他为什么失败，也没有指导他如何改进。现在，ECHO就像给厨师提供了每一步的详细反馈，比如食材的变化、火候的掌握等，让厨师能更好理解厨房的反应，从而不断改进自己的厨艺。这样，厨师不仅学会了怎么做菜，还能预测下一步会发生什么，变得越来越聪明。这个方法让厨师在没有外部高手指导的情况下，也能自己学会做出美味的菜肴，逐渐变成厨房里的大厨。
ELI14 Explained like you're 14
想象你在玩一个游戏，每次你做出一个动作，游戏会告诉你发生了什么，比如“你赢了”或“你输了”。以前，我们只知道结果，但不知道为什么会输，或者下一次怎么做才能赢。现在，这个新方法就像让游戏告诉你每一步的详细情况，比如你用的武器、遇到的怪物、掉落的宝藏。这样，你就能学会根据环境变化调整策略，不再只是盲目试错。就像你在学习骑自行车，不仅知道摔倒了，还知道自己哪里做错了，下一次可以改正。这个方法让AI像你一样，从每次互动中学会理解环境，变得更聪明、更会预测未来的事情。它不仅能帮你在游戏中变强，还能用在机器人、自动驾驶等很多地方，让机器自己学会应对复杂的世界。
Abstract
CLI agents are the closest thing language models have to an embodied setting: the model emits commands, the terminal executes them, and the returned stream -- stdout, errors, files, logs, and traces -- records the consequences. We argue that this stream is a supervision signal, but standard agent RL discards it: GRPO-style training updates action tokens with sparse outcome-level rewards while ignoring environment responses already in the rollout. Failed rollouts provide little policy-gradient signal despite containing rich evidence about how the environment responds. We introduce ECHO (Environment Cross-entropy Hybrid Objective), a hybrid objective that combines the standard policy-gradient loss on action tokens with an auxiliary loss that trains the policy to predict environment observation tokens resulting from its own actions. ECHO reuses the same forward pass as GRPO, requires no additional rollouts, and turns terminal feedback into dense supervision for all rollouts. ECHO doubles GRPO pass@1 on TerminalBench-2.0: Qwen3-8B improves from 2.70% to 5.17%, and Qwen3-14B from 5.17% to 10.79%. ECHO also produces policies that better predict terminal dynamics, even on trajectories they did not generate: across held-out rollouts, it sharply reduces environment-token cross-entropy while GRPO alone barely changes it. From base Qwen3-8B, ECHO matches expert-SFT-then-GRPO performance on held-out terminal tasks without expert demonstrations, and recovers roughly half of the expert-SFT initialization benefit on TerminalBench-2.0. In some settings, the environment prediction loss alone enables verifier-free self-improvement, allowing policies to improve on unseen OOD tasks by learning only from environment interactions. Together, these results suggest that environment observations are not merely context for future actions, but a dense, on-policy supervision signal already present in every rollout.