AutoForge: Automated Environment Synthesis for Agentic Reinforcement Learning
AutoForge automates environment synthesis and ERPO enhances multi-environment RL stability, achieving 73.1 in tau-bench.
Key Findings
Methodology
AutoForge employs tool description documents to automatically generate environment states and function sets, utilizing random walks over dependency graphs to produce diverse tool sequences. These sequences are merged with reasoning nodes and dependency edges to form complex DAGs, serving as task blueprints. The environment-level Relative Policy Optimization (ERPO) algorithm incorporates an LLM-based judge to filter trajectories affected by simulated user errors, improving training stability. Advantage estimation is extended from individual trajectories to the environment level, reducing outlier influence. Experiments on tau-bench, tau2-bench, and VitaBench demonstrate superior performance over baselines, confirming the framework’s effectiveness in environment automation and multi-turn interaction.
Key Results
- AutoForge achieves 73.1 in tau-bench, outperforming Qwen3-Thinking-30B-A3B’s 70.4, with a 2.7 point increase. In out-of-domain ACEBench, RL version improves by 8.5 points, indicating strong generalization. Ablation studies show environment-level advantage estimation and error filtering are critical for performance gains.
- Across multiple environments, AutoForge narrows the gap with closed-source models, approaching industry leaders, validating the multi-environment training approach.
- Filtering erroneous trajectories via the LLM judge stabilizes training, leading to more consistent and robust agent performance.
Significance
This work addresses the bottleneck of environment creation in reinforcement learning, enabling scalable, automated, and diverse environment generation. It tackles the instability caused by simulated user errors and heterogeneity, facilitating more reliable multi-task learning. The proposed ERPO algorithm enhances training stability and efficiency at the environment level, paving the way for more autonomous and adaptable AI agents. By reducing manual effort and expanding task complexity, AutoForge significantly advances the practical deployment of intelligent agents in real-world scenarios, with broad implications for industry and academia.
Technical Contribution
The paper introduces a novel AutoForge framework that automates environment and task generation through tool description-driven processes, combining random walks, reasoning nodes, and dependency edges to create complex task blueprints. The environment-level ERPO algorithm extends advantage estimation to the environment scale, filtering out user error trajectories via an LLM judge, which improves training stability. These innovations enable scalable, diverse environment synthesis and more stable multi-environment reinforcement learning, representing a significant step forward in autonomous agent training.
Novelty
This is the first work to fully automate environment synthesis driven by tool descriptions, integrating complex DAG construction with reasoning nodes and dependency edges. The environment-level advantage estimation and trajectory filtering via LLM judges are novel contributions, addressing instability issues in multi-environment RL. These innovations collectively enable agents to learn in highly diverse, challenging environments with minimal manual intervention, marking a breakthrough in scalable agent training.
Limitations
- The environment synthesis heavily depends on the quality and completeness of tool description documents; poor descriptions limit environment fidelity and diversity.
- In highly dynamic or real-time changing environments, automatic synthesis may not fully capture real-world complexity, leading to potential biases.
- Training in multiple environments incurs high computational costs, requiring significant resources for large-scale deployment. Future work should focus on efficiency improvements and environment realism enhancement.
Future Work
Future directions include integrating multi-modal data (visual, auditory) to enrich environment realism, developing adaptive complexity adjustment mechanisms, and applying meta-learning to improve agents' rapid adaptation to new environments. Additionally, exploring more scalable algorithms for large-scale multi-environment training and extending the framework to real-world deployment scenarios will be critical for advancing autonomous AI agents.
AI Executive Summary
AutoForge introduces a pioneering framework for fully automated environment synthesis tailored for agentic reinforcement learning. By leveraging tool description documents, it constructs environment states and function sets, then employs random walks over dependency graphs to generate diverse, high-difficulty tasks. These tasks are further enriched with reasoning nodes and dependency edges, forming complex DAGs that serve as blueprints for environment creation. The core innovation lies in the environment-level Relative Policy Optimization (ERPO), which extends advantage estimation across multiple environments, coupled with an LLM-based judge that filters trajectories affected by simulated user errors. This combination significantly enhances training stability and efficiency.
Experimental results on tau-bench, tau2-bench, and VitaBench demonstrate that AutoForge outperforms existing models, achieving a score of 73.1 in tau-bench—surpassing the baseline Qwen3-Thinking-30B by 2.7 points—and shows strong out-of-domain generalization, with an 8.5-point improvement on ACEBench. The ablation studies confirm the importance of environment-level advantage estimation and error filtering mechanisms, which contribute to more robust training. The framework effectively narrows the gap between open-source and industry-leading closed models, validating its practical potential.
This work addresses critical challenges in scalable, automated environment creation, enabling agents to learn in more complex, diverse scenarios with minimal manual effort. Its implications extend to industries such as customer service, autonomous planning, and multi-modal AI, where scalable, reliable training environments are essential. Despite current limitations related to environment fidelity and computational costs, future work aims to incorporate multi-modal data, adaptive complexity, and meta-learning for rapid adaptation. Overall, AutoForge marks a significant step toward autonomous, scalable, and robust AI agents capable of tackling real-world complexities.
Deep Analysis
Background
随着大规模语言模型(LLMs)的快速发展，基于强化学习的智能代理在多任务、多场景中展现出巨大潜力。早期工作如ReAct、Toolformer强调工具调用与推理结合，但多依赖人工环境设计，难以实现大规模自动化。近年来，自动环境合成成为研究热点，旨在降低成本、提升多样性。现有方法如ToolSandbox、τ-bench等，虽能模拟环境，但多依赖手工标注或有限自动化，难以满足复杂多变的需求。AutoForge通过工具描述文档驱动的自动合成环境，突破了人工依赖瓶颈，为代理自主学习提供新平台。
Core Problem
当前环境合成方法存在自动化程度不足、环境多样性有限、模拟用户行为不稳定等问题。人工设计环境费时费力，难以快速扩展到复杂场景。模拟用户的不稳定行为会引入偏差，影响训练效果。此外，单一环境的RL训练效率低，难以应对多任务和多轮交互需求。这些问题限制了代理在真实复杂环境中的应用，亟需一种高效、自动、稳定的环境合成方案。
Innovation
AutoForge的创新点包括：1）工具描述文档驱动的环境自动构建，减少人工干预；2）随机游走结合推理节点和依赖边，生成复杂任务蓝图，提升任务多样性；3）引入环境层相对优势估计(ERPO)，增强多环境训练的稳定性；4）利用LLM作为评判机制，过滤模拟用户中的错误轨迹，确保训练质量。这些创新结合自动化、推理和多环境优化，突破了传统手工设计和单环境训练的局限，为代理自主学习提供了强大支撑。
Methodology
- �� 利用工具描述文档自动生成环境状态结构和功能集。
- �� 通过随机游走在工具依赖图中采样多样工具序列，增强任务复杂性。
- �� 合并工具序列，插入推理节点和依赖边，构建复杂DAG蓝图。
- �� 根据DAG生成高难度任务，模拟用户行为，生成环境初始状态和任务意图。
- �� 执行工具调用，获得最终环境状态，评估任务完成情况。
- �� 利用LLM过滤用户错误轨迹，提升训练稳定性。
- �� 引入环境级优势估计，减少异常值影响，优化多环境RL训练。
Experiments
在tau-bench、tau2-bench和VitaBench上，采用Qwen3-Thinking-235B作为环境合成模型，生成1078个高难度任务环境。使用GPT-4.1模拟用户，训练参数包括批次32、每个样本8条轨迹。对比基线模型，AutoForge在性能指标上显著优越，验证了环境自动合成和多轮交互的有效性。还进行了消融实验，验证环境层优势估计和错误过滤机制的贡献。
Results
AutoForge在tau-bench中达成73.1分，优于Qwen3-Thinking-30B-A3B的70.4，提升3个百分点。在out-of-domain ACEBench中，RL版本提升8.5点，表现出强泛化能力。消融分析显示，环境层优势估计和错误过滤机制对性能提升关键，去除错误轨迹后训练更稳定，效果更佳。整体结果验证了自动环境合成的有效性和训练的鲁棒性。
Applications
该方法可广泛应用于智能客服、自动化问答、任务规划等场景，尤其适合需要多轮交互和复杂推理的任务。自动环境生成降低了人工成本，加快了模型的训练速度，提升了多任务适应能力。未来可结合多模态信息，拓展到视觉、声音等多感知场景，推动自主智能体的产业化应用。
Limitations & Outlook
当前环境合成依赖于高质量的工具描述文档，描述不足会影响环境真实性。复杂动态环境中，自动合成可能偏离真实场景，存在偏差。算法在大规模多环境训练中计算成本高，需优化效率。未来需增强环境多样性和真实性，降低成本，提升适应性。
Plain Language Accessible to non-experts
想象你在一家厨房做菜，环境就是厨房的布局和所有的工具，比如锅、刀、菜板。每次做菜前，你需要准备好所有工具和食材（环境状态），然后按照菜谱（任务）逐步操作。AutoForge就像是一个智能厨师，它能自动设计出不同的厨房布局和菜谱，还能根据不同的食材和工具组合出各种新菜。它用一种叫“工具描述文档”的说明书，告诉自己厨房里有哪些工具，然后随机组合，像玩拼图一样拼出复杂的菜谱。为了确保菜做得好，它还会用一个“评判员”——类似厨师的朋友，检查每一步是否正确，过滤掉错误的操作。这样，厨师（代理）就能在不同的厨房里快速学会做各种菜，不怕出错，也能做出更复杂的菜。这种系统让厨房变得更智能、更灵活，也让我们学做菜变得更容易、更有趣。
ELI14 Explained like you're 14
想象你在学校里学做菜，老师会给你一份菜谱和一些工具，比如锅、刀、菜板。你需要按照菜谱一步步操作，把食材变成一道菜。AutoForge就像一个超级聪明的厨师，它可以自己设计不同的厨房和菜谱，不用老师手工帮忙。它会用说明书告诉自己有哪些工具，然后随机拼出各种复杂的菜谱，比如先切菜，再炒菜，最后装盘。为了确保每一步都正确，它还会请一个“评判员”——就像厨师的朋友，检查你是不是做错了，帮你过滤掉错误操作。这样，厨师就能在不同的厨房里快速学会做很多菜，而且还能做出特别复杂的菜。这个系统让学做菜变得更简单、更有趣，也能帮厨师变得更厉害。
Abstract
Conducting reinforcement learning (RL) in simulated environments offers a cost-effective and highly scalable way to enhance language-based agents. However, previous work has been limited to semi-automated environment synthesis or tasks lacking sufficient difficulty, offering little breadth or depth. In addition, the instability of simulated users integrated into these environments, along with the heterogeneity across simulated environments, poses further challenges for agentic RL. In this work, we propose: (1) a unified pipeline for automated and scalable synthesis of simulated environments associated with high-difficulty but easily verifiable tasks; and (2) an environment level RL algorithm that not only effectively mitigates user instability but also performs advantage estimation at the environment level, thereby improving training efficiency and stability. Comprehensive evaluations on agentic benchmarks, including tau-bench, tau2-Bench, and VitaBench, validate the effectiveness of our proposed method. Further in-depth analyses underscore its out-of-domain generalization.