Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks
Proposes Plan-and-Act framework with synthetic data augmentation, achieving 57.58% success in long-horizon web tasks.
Key Findings
Methodology
The approach employs a dual-model architecture: a Planner (based on GPT-4) generates structured high-level plans, while an Executor translates these plans into environment-specific actions. Synthetic data generation involves analyzing successful trajectories with a teacher LLM, reverse-engineering structured plans grounded in actual actions, and expanding training data via diverse sampling. Grounded plan generation ensures plans align with real execution, while synthetic plan expansion addresses data scarcity. The models are trained with reinforcement of successful trajectories filtered by an ORM reward model, enabling robust performance in web navigation benchmarks. The framework supports dynamic replanning and chain-of-thought reasoning, further enhancing robustness.
Key Results
- On WebArena-Lite, the model achieved a success rate of 57.58%, surpassing prior methods (~50%), demonstrating improved long-term planning capabilities.
- On WebVoyager's text-only tasks, success rate reached 81.36%, significantly higher than baseline (~70%), validating the effectiveness of synthetic data augmentation.
- Ablation studies confirmed that dynamic replanning and chain-of-thought reasoning contributed approximately 5% and 3% improvements respectively, confirming their roles in performance gains.
Significance
This work addresses fundamental challenges in long-horizon planning with large language models, offering a scalable, data-efficient framework that significantly improves success rates in web navigation tasks. By integrating synthetic data generation with a hierarchical planning architecture, it overcomes data scarcity and environmental variability issues. The approach advances the state-of-the-art in autonomous agents, providing a foundation for deploying more reliable, adaptable AI systems in real-world applications such as web automation, robotics, and digital assistants. Its capacity for dynamic replanning and reasoning makes it particularly suitable for complex, unpredictable environments, marking a substantial step forward in AI autonomy.
Technical Contribution
The core technical innovation is the combination of a dual-model architecture with a novel synthetic data pipeline. Grounded plan generation leverages trajectory analysis to produce high-quality training data without manual annotation. Synthetic plan expansion scales this process, generating large, diverse datasets for training the Planner. The models incorporate chain-of-thought prompting and dynamic replanning, enabling better handling of environment changes. These contributions collectively improve long-horizon task performance, offering a scalable, generalizable solution for complex decision-making tasks.
Novelty
This is the first comprehensive framework integrating synthetic data generation with hierarchical planning for large language model-based agents in web navigation. Unlike prior work that relies solely on prompt engineering or reinforcement learning, this approach systematically creates high-quality training data, grounded in real trajectories, and supports dynamic adjustment. Its simplicity, scalability, and effectiveness in surpassing SOTA benchmarks highlight its novelty in the field of long-term autonomous planning.
Limitations
- Despite improvements, the models still struggle with highly novel or extremely complex environments where environment understanding is limited. The reliance on synthetic data quality can introduce biases, affecting generalization.
- Training costs are substantial due to large-scale data generation and model fine-tuning, which may limit deployment in resource-constrained settings.
- Dynamic replanning, while effective, increases computational overhead and complexity, potentially impacting real-time applications.
Future Work
Future directions include integrating multi-modal inputs such as vision and speech to enrich environment understanding, employing reinforcement learning to optimize decision policies, and developing more efficient synthetic data generation algorithms. Expanding the framework to other domains like robotics and industrial automation, and improving model robustness against unseen environments, are also promising avenues.
AI Executive Summary
Long-horizon, multi-step tasks pose a significant challenge for autonomous agents, especially when relying solely on large language models (LLMs). Traditional approaches often struggle with maintaining coherent plans over multiple steps, adapting to environmental changes, and generalizing to unseen scenarios. This paper introduces the Plan-and-Act framework, which explicitly separates high-level planning from low-level execution, addressing these issues through a hierarchical architecture. The Planner generates structured, high-level plans based on user goals, while the Executor translates these plans into environment-specific actions, such as clicking or typing in web navigation tasks.
A key innovation is the synthetic data generation pipeline, which leverages LLMs to analyze successful trajectories, reverse-engineer grounded plans, and expand training datasets via diverse sampling. This approach overcomes the scarcity of annotated planning data, enabling the models to learn more accurate and generalizable strategies. The framework incorporates dynamic replanning, allowing the Planner to update plans after each step based on environment feedback, and chain-of-thought reasoning to improve decision quality.
Experimental results on WebArena-Lite and WebVoyager benchmarks demonstrate the effectiveness of the approach. The model achieved a success rate of 57.58% on WebArena-Lite, surpassing previous methods, and 81.36% on WebVoyager's text-only tasks. Ablation studies confirmed that synthetic data, dynamic replanning, and reasoning significantly contribute to performance gains. These findings suggest that the proposed framework offers a scalable, robust solution for complex autonomous tasks, with broad implications for AI in web automation, robotics, and beyond.
Despite these advances, challenges remain in handling highly novel environments and reducing training costs. Future work will focus on multi-modal integration, reinforcement learning, and more efficient data synthesis, aiming to develop even more capable, adaptable autonomous agents.
Deep Analysis
Background
近年来，随着GPT-4等大模型在自然语言处理中的突破，推动了自主智能体的发展。早期工作如ReAct、WebGPT结合推理与行动，实现了在简单任务中的成功，但在长远、多步骤任务中仍存在规划、记忆和环境适应的瓶颈。尤其是在Web导航、机器人控制等复杂场景中，单一模型难以兼顾策略制定与具体执行，导致任务失败率升高。近年来，分层规划、强化学习等技术被引入，但仍受限于数据稀缺、环境动态变化和模型泛化能力不足的问题。本文在此背景下，提出结合合成数据增强的“计划与行动”框架，旨在突破长远任务中的瓶颈。
Core Problem
长远、多步骤任务中的规划难题是AI研究的核心难点。现有方法多依赖单一模型，难以同时兼顾高层策略和低层操作，导致任务中断或偏离目标。环境的动态变化使得模型难以实时调整计划，缺乏高质量训练数据更是限制因素。手工标注成本高、难以覆盖所有场景。如何设计一个既能高效规划，又能适应环境变化的系统，是亟待解决的问题。
Innovation
本研究的创新包括：1）引入双模型架构，将规划与执行职责分离，提升长远任务中的决策效率；2）开发合成数据生成流程，通过轨迹反向工程和多样化样本扩充，缓解数据稀缺问题；3）提出 grounded plan generation，利用轨迹分析自动生成高质量结构化计划；4）采用 synthetic plan expansion，通过大规模样本扩充增强模型泛化能力。这些创新显著提升了Web导航任务中的成功率，模型支持动态重规划和链式推理，增强了环境适应性。
Methodology
- �� 设计双模型架构，包括基于GPT-4的规划器（Planner）和执行器（Executor）。
- �� 规划器接收用户请求，生成结构化高层计划，指导执行器操作。
- �� 执行器将计划转化为具体环境动作（如网页点击、输入），并反馈环境状态。
- �� 利用轨迹反向工程技术，自动生成 grounded plan，确保计划与实际轨迹一致。
- �� 采用合成数据扩充策略，通过随机采样和多样化样本生成大量训练数据。
- �� 使用ORM模型筛选成功轨迹，保证数据质量。
- �� 支持动态重规划，每步执行后更新计划，适应环境变化。
- �� 引入链式推理（CoT）增强推理能力，提升复杂任务表现。
Experiments
在WebArena-Lite和WebVoyager两个公开基准上进行评估。采用成功率作为主要指标，比较不同模型和数据增强策略的效果。训练中利用大规模合成数据，调优模型参数，进行消融实验验证各技术贡献。通过与单模型、无合成数据、静态计划等对比，验证框架的有效性。实验结果显示，模型在WebArena-Lite达57.58%的成功率，在WebVoyager文本任务中达81.36%，显著优于未增强模型。
Results
模型在WebArena-Lite上实现了57.58%的成功率，超越之前的50.2%，验证了合成数据和分层规划的有效性。在WebVoyager文本任务中，成功率达81.36%，优于未使用合成数据的模型（约70%）。消融实验显示，动态重规划和链式推理分别提升了成功率约5%和3%，增强了模型的鲁棒性和适应性。这些结果表明，所提出的方法在复杂长远任务中具有显著优势。
Applications
该框架适用于Web自动化、智能助理、机器人控制等场景，特别适合需要多步骤决策和环境交互的任务。实现条件包括高质量的环境轨迹数据和模型微调能力。未来可结合多模态信息（如视觉、语音）扩展应用范围，推动自主系统在实际复杂环境中的部署。
Limitations & Outlook
模型在极端复杂或未见过的环境中仍存在失败风险，主要由于环境理解和泛化能力不足。合成数据的质量受预训练模型限制，可能引入偏差。训练成本较高，尤其在大规模数据生成和微调阶段，限制了其在资源有限环境中的应用。未来需优化数据生成效率和模型泛化能力，以应对更复杂的实际场景。
Plain Language Accessible to non-experts
想象你在厨房做饭。每次做菜都需要先想好步骤，比如洗菜、切菜、炒菜。这个过程很像给机器人规划任务：先制定一个详细的菜单（计划），然后逐步操作（执行）。但有时候，厨房里的情况会变，比如突然发现没有某种调料，原来的计划就得改。为了让机器人更聪明，我们教它不仅能制定菜谱，还能根据厨房的实际情况随时调整。我们还用模拟的厨房环境，生成各种可能的场景，让机器人学会应对不同的突发情况。这样，它就能在真实厨房中灵活应对各种挑战，做出美味佳肴。这个方法让机器人变得更聪明、更可靠，也更像一个会变通的厨师。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，比如冒险游戏。你需要先想好怎么完成任务，比如找到宝藏、打败怪兽，然后一步步行动。有时候，游戏中的情况会突然改变，比如怪兽突然出现或者通道被堵住，你得马上想办法调整策略。这就像让一个机器人学会自己规划路线和行动，但它之前从没遇到过这些突发情况。为了帮它变得更聪明，我们用模拟的游戏场景，创造各种不同的挑战，让它练习应对。我们还教它如何根据新情况调整计划，就像你在游戏中临场发挥一样。这样，机器人就能在真实世界的复杂任务中表现得更好，更像一个聪明的冒险者。
Abstract
Large language models (LLMs) have shown remarkable advancements in enabling language agents to tackle simple tasks. However, applying them for complex, multi-step, long-horizon tasks remains a challenge. Recent work have found success by separating high-level planning from low-level execution, which enables the model to effectively balance high-level planning objectives and low-level execution details. However, generating accurate plans remains difficult since LLMs are not inherently trained for this task. To address this, we propose Plan-and-Act, a novel framework that incorporates explicit planning into LLM-based agents and introduces a scalable method to enhance plan generation through a novel synthetic data generation method. Plan-and-Act consists of a Planner model which generates structured, high-level plans to achieve user goals, and an Executor model that translates these plans into environment-specific actions. To train the Planner effectively, we introduce a synthetic data generation method that annotates ground-truth trajectories with feasible plans, augmented with diverse and extensive examples to enhance generalization. We evaluate Plan-and-Act using web navigation as a representative long-horizon planning environment, demonstrating a state-of-the-art 57.58% success rate on the WebArena-Lite benchmark as well as a text-only state-of-the-art 81.36% success rate on WebVoyager.