Does The Way You Plan Matter? An Empirical Study of Planning Representations for LLM Web Agents
Introduces PLANAHEAD framework, systematically evaluating natural language plan representations' impact on multimodal LLM web agent performance.
Key Findings
Methodology
Using an automated difficulty grading pipeline based on WebArena, the study compares four plan formats—sequential subgoals, narrative, pseudocode, checklist—across models like GPT-4, Qwen, Gemini. It employs the Achievement Rate (AR) and Solved-Task Consistency (STC) metrics, capturing success probability and robustness over multiple stochastic trials. The static planner-executor framework ensures consistent evaluation, analyzing how plan form and model roles influence task success on challenging tasks, especially in the hard subset.
Key Results
- Model preferences for plan formats vary: GPT-4 favors narrative, Qwen prefers checklists, Gemini performs well across formats. Static planning outperforms dynamic in hard tasks, achieving AR up to 66% and STC over 80%, indicating plan form critically affects robustness.
- Separating planning and execution modules allows leveraging strengths of different models, with combinations like GPT-4 as planner and Gemini as executor yielding 15% higher success. The new metrics AR and STC provide nuanced insights into task achievability and stability.
- Findings highlight that plan formulation and model role division significantly influence web agent robustness, guiding future modular system designs for improved reliability in complex environments.
Significance
This work advances understanding of how diverse natural language plan representations affect large multimodal models in web navigation. It challenges the traditional sequential subgoal paradigm, demonstrating that alternative formats can enhance performance and robustness. The introduced evaluation metrics enable more precise assessment of agent stability, informing the design of more reliable autonomous web systems. Overall, it bridges the gap between planning theory and practical deployment, fostering development of adaptable, resilient AI agents for real-world applications.
Technical Contribution
The study's core innovations include: (1) automated task difficulty classification ensuring reproducibility; (2) systematic comparison of four natural language plan formats; (3) development of AR and STC metrics capturing success probability and consistency over multiple runs. It integrates these into a static planning framework, revealing how plan form and model roles influence performance, and provides a comprehensive evaluation methodology for future research.
Novelty
This is the first systematic comparison of multiple natural language plan expressions in the context of multimodal LLM web agents. It introduces the AR and STC metrics, emphasizing the importance of plan robustness and variability, which surpasses traditional success rate measures. The work highlights the impact of plan representation on agent performance, filling a critical research gap and offering new insights into planning strategies for complex web tasks.
Limitations
- The evaluation is limited to WebArena and three models, restricting generalization. Real-world web environments are more dynamic, requiring further validation.
- Static planning, while effective in experiments, may not adapt well to rapidly changing web contexts, necessitating integration with dynamic planning.
- Plan representations, though diverse, may still lack expressiveness for extremely complex or ambiguous tasks, indicating room for richer, hybrid formats.
Future Work
Future directions include exploring hybrid plan formats combining natural language with graphical or structured representations, integrating adaptive dynamic planning strategies, and testing across broader datasets and models. Developing more sophisticated evaluation metrics and real-time adaptive systems will further enhance the robustness and applicability of web agents in unpredictable environments.
AI Executive Summary
Recent advances in large multimodal language models have significantly improved autonomous web agents' capabilities. Yet, challenges remain in exploration, key step omission, and task constraint sensitivity, especially in complex, multi-step tasks. Traditional planning approaches rely on sequential subgoal natural language descriptions, which may limit flexibility and robustness. This study introduces PLANAHEAD, a static planner-executor framework, to systematically evaluate how different natural language plan representations influence agent performance.
Using the WebArena benchmark, the authors automatically categorize tasks into easy, medium, and hard levels, focusing on the most challenging subset. They compare four plan formats—sequential subgoals, narrative, pseudocode, and checklists—across models like GPT-4, Qwen, and Gemini. To account for stochastic variability, they propose two novel metrics: Achievement Rate (AR), measuring success probability over multiple runs, and Solved-Task Consistency (STC), assessing performance stability.
Results reveal that plan form and model roles significantly impact robustness and success. For instance, GPT-4 excels with narrative plans, while Qwen prefers checklists. Static planning generally outperforms dynamic approaches in hard tasks, achieving success rates up to 66%. The combination of models—such as GPT-4 as planner and Gemini as executor—further boosts performance. These findings demonstrate that flexible plan representations and modular architectures are crucial for resilient web agents.
The introduced metrics provide a nuanced understanding of task achievability and stability, surpassing traditional success rate measures. Despite promising results, limitations include dataset scope and environment complexity. Future work aims to explore hybrid plan formats, adaptive dynamic planning, and broader evaluations, advancing autonomous web agents toward higher reliability and real-world applicability.
Deep Analysis
Background
Web agents已从规则驱动逐步演变到深度学习和大模型，代表性工作如WebGPT、WebAgent、MindWeb推动了自主导航和任务完成能力的提升。然而，复杂多步骤任务中的探索能力仍不足，容易遗漏关键步骤或陷入循环。规划作为提升执行效率的关键策略逐渐受到关注，但现有研究多集中于自然语言子目标的顺序拆解，缺乏对多样表达形式的系统比较。缺少统一评估指标，限制了规划策略的优化空间。随着多模态模型的发展，如何设计高效、鲁棒的计划表达成为研究热点。
Core Problem
尽管规划在Web代理中扮演重要角色，但不同自然语言表达方式对模型性能的影响尚未充分理解。传统方法多采用顺序子目标，忽视表达的多样性和信息丰富性。缺乏科学的任务难度划分机制，难以在不同复杂度任务中评估规划策略的优劣。现有指标多偏重成功率，未能反映多次试验中的稳定性和鲁棒性。这些问题限制了Web代理在实际环境中的应用效果，亟需引入多样化表达和更科学的评估体系。
Innovation
本研究的创新点包括：1）引入自动任务难度划分机制，确保评估的客观性；2）提出多种自然语言计划表达（叙述、伪代码、清单），丰富表达形式，探索其对模型性能的影响；3）设计AR和STC指标，全面衡量任务达成率和鲁棒性，超越传统成功率指标。这些创新结合静态规划框架，系统分析不同模型在硬任务中的表现差异，为多模态Web代理的设计提供理论支持和实践路径。
Methodology
- �� 任务选择：基于WebArena，筛选出381个任务，采用BrowserGym的通用代理进行自动难度划分，标记Easy、Medium、Hard。
- �� 任务划分：用5个不同LLM模型进行多次试验（N=5），成功率为100%的为Easy，全部失败的为Hard，其余为Medium。
- �� 规划框架：采用静态PLANAHEAD框架，LLM作为规划者，生成一次完整计划（顺序子目标、叙述、伪代码或清单），然后由执行器模型在每步执行。
- �� 计划表达：分别用四种不同格式（顺序子目标、需求清单、伪代码、叙述）生成计划，控制模型温度以确保多样性。
- �� 评估指标：提出Achievement Rate（AR）衡量多次试验中任务的达成情况，Solving-Task Consistency（STC）衡量多次试验中的表现稳定性。
- �� 实验设计：在硬任务集上，测试不同模型（GPT-4、Qwen、Gemini）和不同计划表达方式的性能，比较静态与动态规划的效果，分析模型偏好与鲁棒性差异。
Experiments
在WebArena硬任务集上，N=5次多次试验，评估不同模型和计划表达的性能。通过AR和STC指标，分析模型偏好、表达形式影响，验证静态优于动态的结论。所有试验在相同硬件环境下进行，确保结果的可比性。实验还包括模型组合的性能比较，揭示模型角色分工的优势。对比传统成功率（SR），新指标更细腻反映鲁棒性和稳定性。
Results
不同模型对计划表达的偏好差异明显：GPT-4偏好叙述式，Qwen偏好清单式，Gemini表现均衡。静态规划在硬任务中成功率达66%，AR最高达66%，STC超过80%，优于动态规划。模型组合（如GPT-4作为规划者，Gemini作为执行者）成功率提升15%。AR和STC指标能更准确反映任务达成和鲁棒性，为Web代理设计提供新评估标准。这些结果验证了多样化计划表达和模型角色分工的重要性。
Applications
该研究为自主Web代理的设计提供理论基础和实践指南，适用于智能客服、自动化数据采集、电子商务等场景。通过优化计划表达和模型组合，可显著提升任务完成率和鲁棒性，满足复杂环境下的自主决策需求。未来，结合动态规划和多模态表达，将推动Web代理在实际应用中的广泛部署。
Limitations & Outlook
研究仅在WebArena和少数模型上进行，泛化能力有限。静态规划在动态环境中可能表现不足，未来需结合动态策略。计划表达虽多样，但在极端复杂任务中仍存在信息不足或表达不充分的问题。模型间的协同优化也需进一步探索，提升整体适应性和效率。
Plain Language Accessible to non-experts
想象你在厨房做饭。每次做饭都需要准备食材、按照步骤操作、检查味道。不同的厨师可能用不同的方式描述步骤，比如写成详细的菜谱、讲故事、列清单或用伪代码。这个研究就像在厨房里试验不同的描述方式，看哪种能帮厨师更快、更好地做出美味菜肴。通过比较这些不同的描述，发现某些表达更适合特定厨师或菜肴。最终目标是让厨房里的每个人都能用最合适的方式，做出最棒的菜。这就像让AI学会用不同的“菜谱”来完成任务，确保它们在复杂环境中都能顺利完成。
ELI14 Explained like you're 14
想象你在学校里做科学实验。老师给你一个任务，比如“用不同材料建一个桥”。你可以写详细的步骤、讲一个故事、列出材料清单，或者写伪代码告诉自己怎么做。这个研究就像在试验用不同的“说明书”帮助AI完成网页任务。科学家发现，有时候用讲故事的方式能让AI更聪明地理解任务，有时候用清单更快。通过这些不同的“说明书”，他们找到最适合AI的表达方法，让它在复杂的网页任务中也能顺利完成。就像你用不同的方法学会做事情，最后找到最适合自己的那一种！
Abstract
Despite recent advances, LLM-based web agents still struggle with limited exploration, omission of critical steps, and sensitivity to task constraints. Prior work suggests that many of these failures stem from weaknesses in planning, yet the impact of alternative natural language plan representation remains unexplored. To address this, we introduce PlanAhead, a static planner-executor framework that evaluates the impact of plan representation in agent performance. We first automatically categorize WebArena tasks into 3 difficulty levels, enabling consistent difficulty grading without human annotation. Then we systematically evaluate 4 different plan representations on the tasks categorized as hard: sequential subgoals, narrative, pseudocode, and checklist; across different families of multimodal LLM powered agents (OpenAI, Alibaba, and Google). To account for stochastic variability, we introduce two novel evaluation metrics: Achievement Rate (AR) and Solved-Task Consistency (STC). Our results show that both, the plan formulation and the underlying LLM generating the plan, significantly influence web-agent robustness and task success.