Agentic Entropy-Balanced Policy Optimization
AEPO introduces dynamic entropy balancing and gradient regulation, enhancing stability and exploration in multi-turn web agent RL with only 1K samples.
Key Findings
Methodology
AEPO combines a dynamic entropy-balanced rollout mechanism with entropy-aware policy optimization. The former adaptively allocates global and branch sampling based on entropy pre-monitoring, employing penalties to prevent over-branching. The latter integrates stop-gradient operations into the high-entropy clipping term, preserving and rescaling gradients of uncertain tokens, and uses entropy-aware advantage estimation to prioritize learning on high-uncertainty tokens. These components collectively improve exploration diversity and training stability, addressing high entropy-induced collapse.
Key Results
- Across 14 datasets, AEPO outperforms 7 mainstream RL algorithms. With only 1K samples, Qwen3-14B with AEPO achieves 47.6% Pass@1 on GAIA, 11.2% on Humanity’s Last Exam, and 43.0% on WebWalkerQA; for Pass@5, it reaches 65.0%, 26.0%, and 70% respectively. Results demonstrate enhanced sampling diversity and stable policy entropy, facilitating scalable web agent training.
- Comparative analysis shows AEPO's superiority in exploration diversity and policy stability. Ablation studies confirm that entropy pre-monitoring and stop-gradient significantly contribute to performance gains, effectively mitigating over-branching and gradient clipping issues.
- The experiments validate that AEPO maintains high entropy during training, improves exploration in high-uncertainty states, and achieves consistent gains across tasks, indicating its potential for general web agent development.
Significance
This work addresses fundamental challenges in reinforcement learning for web agents, notably the high entropy collapse and gradient clipping issues. By proposing a principled entropy balancing framework, AEPO enables more robust and scalable training of multi-turn, tool-using agents. Its ability to sustain exploration diversity while ensuring training stability paves the way for more autonomous, intelligent systems capable of complex reasoning and long-horizon planning. The methodology offers a new paradigm for RL in dynamic, uncertain environments, with broad implications for AI research and industry applications.
Technical Contribution
AEPO's core innovations include a dynamic entropy pre-monitoring mechanism for adaptive resource allocation, a penalty strategy to prevent over-branching, and a stop-gradient integrated into the high-entropy clipping process to preserve gradient signals. The entropy-aware advantage estimation further refines learning focus on uncertain tokens. These contributions collectively provide a novel, theoretically grounded framework that balances exploration and stability, surpassing traditional entropy maximization or fixed clipping approaches, and opening new avenues for scalable, stable RL training.
Novelty
This is the first systematic study of high-entropy collapse and gradient clipping issues in web agent RL, proposing a unified solution that dynamically balances exploration and stability. Unlike prior work focusing solely on entropy maximization or static gradient clipping, AEPO introduces a flexible, adaptive framework with theoretical justification. Its combination of entropy pre-monitoring, penalized branching, and stop-gradient regulation represents a significant step forward in RL research, with broad applicability beyond web agents.
Limitations
- Despite improvements, AEPO may still face challenges in extremely high-entropy states where gradients become unstable, requiring further tuning or adaptive mechanisms.
- The additional computational overhead from entropy monitoring and dynamic resource allocation may limit real-time deployment in large-scale systems.
- Current validation is primarily on web reasoning tasks; applicability to other RL domains with different dynamics remains to be explored.
Future Work
Future research will focus on integrating meta-learning to automate parameter tuning, extending the framework to multi-modal and multi-task scenarios, and optimizing computational efficiency. Exploring adaptive mechanisms for even more robust entropy control, as well as broader validation across diverse RL environments, will be key to advancing scalable, autonomous AI agents.
AI Executive Summary
Recent advances in large language models (LLMs) have revolutionized natural language reasoning, but deploying them as web agents involves complex multi-turn interactions with tools and search engines. Traditional reinforcement learning (RL) approaches often struggle with high-entropy states, leading to training collapse and limited exploration. This paper introduces Agentic Entropy-Balanced Policy Optimization (AEPO), a novel RL algorithm designed to address these challenges through a combination of dynamic entropy management and gradient regulation.
AEPO's key innovation lies in its two core components. First, the dynamic entropy-balanced rollout mechanism adaptively allocates sampling resources based on real-time entropy monitoring, preventing over-branching in high-uncertainty states. Second, the entropy-balanced policy optimization employs a stop-gradient operation during policy updates, ensuring valuable exploration signals from high-entropy tokens are preserved and properly scaled. These mechanisms work synergistically to maintain exploration diversity while stabilizing training.
Extensive experiments across 14 challenging datasets demonstrate AEPO's superior performance over seven mainstream RL algorithms. Notably, with only 1K samples, the Qwen3-14B model equipped with AEPO achieves 47.6% Pass@1 on GAIA, outperforming baselines by significant margins. The results highlight the algorithm's ability to sustain high policy entropy, enhance exploration, and prevent collapse, thus enabling scalable training of web agents.
This work offers a breakthrough in RL for complex, uncertain environments. By systematically addressing high-entropy challenges, AEPO paves the way for more autonomous, reliable, and versatile AI agents capable of long-horizon reasoning and multi-tool integration. Its theoretical and practical innovations provide a foundation for future research in scalable, stable reinforcement learning, with broad implications for AI deployment in real-world scenarios.
Deep Dive
Glossary
Entropy (熵)
衡量模型输出不确定性程度的指标，数值越高代表越不确定。In this paper, used to guide exploration and调节采样策略。
用于动态调节采样资源和策略更新中的梯度调控。
Stop-gradient (停止梯度)
在反向传播中阻断梯度流的技术，确保某些参数在训练中不被更新。用于高熵Token的梯度调节。
在策略优化中应用，保持探索信号的传递。
熵预监控
在训练前实时监测模型输出的熵值，用于动态分配采样资源。
AEPO中用以避免过度分支和训练崩溃。
分支惩罚
对连续高熵分支施加惩罚，限制过度探索。
在滚动采样中实现多样性平衡。
熵优势估计
结合熵信息调整优势估计，优先学习不确定性高的Token。
提升探索效率和模型泛化能力。
Open Questions Unanswered questions from this research
- 1 如何在极端高熵环境中进一步稳定训练，仍是未解难题。未来需探索更鲁棒的熵调节机制，以适应更复杂场景。
- 2 算法在大规模模型和多任务环境中的适应性和效率仍需验证，特别是在实际工业应用中。
Abstract
Recently, Agentic Reinforcement Learning (Agentic RL) has made significant progress in incentivizing the multi-turn, long-horizon tool-use capabilities of web agents. While mainstream agentic RL algorithms autonomously explore high-uncertainty tool-call steps under the guidance of entropy, excessive reliance on entropy signals can impose further constraints, leading to the training collapse. In this paper, we delve into the challenges caused by entropy and propose the Agentic Entropy-Balanced Policy Optimization (AEPO), an agentic RL algorithm designed to balance entropy in both the rollout and policy update phases. AEPO comprises two core components: (1) a dynamic entropy-balanced rollout mechanism that adaptively allocate global and branch sampling budget through entropy pre-monitoring, while imposing a branch penalty on consecutive high-entropy tool-call steps to prevent over-branching issues; and (2) Entropy-Balanced Policy Optimization that inserts a stop-gradient operation into the high-entropy clipping term to preserve and properly rescale gradients on high-entropy tokens, while incorporating entropy-aware advantage estimation to prioritize learning on high-uncertainty tokens. Results across 14 challenging datasets show that AEPO consistently outperforms 7 mainstream RL algorithms. With just 1K RL samples, Qwen3-14B with AEPO achieves impressive results: 47.6% on GAIA, 11.2% on Humanity's Last Exam, and 43.0% on WebWalker for Pass@1; 65.0% on GAIA, 26.0% on Humanity's Last Exam, and 70.0% on WebWalker for Pass@5. Further analysis reveals that AEPO improves rollout sampling diversity while maintaining stable policy entropy, facilitating scalable web agent training.