Plover: Steering GUI Agents through Plan-Centric Interaction
Plover introduces a plan-centric vision-based GUI automation system with externalized, inspectable, and editable task plans, enhancing transparency and control.
Key Findings
Methodology
Plover employs a planner–executor architecture, externalizing task plans as persistent, versioned structures. The planner, based on heuristic search and rule-based methods, generates explicit structured plans. The executor grounds each step into concrete mouse, keyboard, and pixel-based actions, providing real-time status feedback. The system introduces an intelligent replanning (IR) mechanism, combining user-driven and system-driven modes, to dynamically revise pending steps while preserving execution history. Multi-modal interactions, including natural language and screenshot annotations, facilitate precise, localized corrections, significantly improving system transparency and repairability.
Key Results
- In 38 challenging GUI tasks, initial autonomous execution failed in 26 cases. After collaborative repair using Plover, success rate increased to 88%, with an average of 2.04 interventions per task. The system effectively identified failure modes like spatial ambiguity and multi-step drift, with localized plan edits significantly restoring task completion. These results demonstrate the efficacy of plan externalization and multi-modal correction in enhancing GUI repair success.
- Scenario-based stability analysis confirmed that exposing plans and enabling visual interventions improved task continuity and user trust. Compared to traditional autonomous systems, Plover achieved higher repair success and user satisfaction, validating the approach's practical benefits.
- Across diverse GUI environments, the system's ability to identify and repair failures via localized plan edits and multimodal inputs proved robust, especially in complex, dense interfaces where traditional methods struggle.
Significance
This work addresses fundamental limitations of autonomous GUI agents by externalizing task plans, enabling explicit supervision, and facilitating targeted repairs. It advances the field toward more transparent, controllable, and adaptable automation, crucial for high-stakes workflows. The approach bridges the gap between fully autonomous systems and human-in-the-loop interaction, fostering trust and reliability. The plan-centric design paves the way for future integration of deep learning and multimodal inputs, promising more intelligent and resilient GUI automation solutions.
Technical Contribution
The paper introduces a novel architecture that externalizes task plans as persistent, editable artifacts, enabling explicit supervision and repair. The integration of an IR mechanism allows dynamic, localized plan revisions, maintaining task continuity. The system supports multimodal interactions—natural language and visual annotations—enhancing correction precision. This framework departs from prior black-box models by providing transparent, inspectable plans that facilitate human oversight and iterative repair, representing a significant step forward in GUI automation research.
Novelty
This is the first work to treat task plans as persistent, external artifacts in GUI automation, enabling real-time inspection, editing, and localized repair during execution. Unlike prior systems that rely solely on internal reasoning or static scripts, Plover emphasizes explicit plan visualization and multimodal correction, fundamentally enhancing system transparency and user control.
Limitations
- In highly dynamic or structurally unstable interfaces, plan accuracy and repair effectiveness diminish, especially when interface changes outpace plan updates.
- Multimodal correction depends heavily on user input quality; misannotations or misunderstandings can lead to ineffective repairs.
- Performance in large-scale, high-frequency correction scenarios needs further optimization to reduce latency and improve responsiveness.
Future Work
Future research will focus on integrating deep learning models for improved scene understanding and plan generation, expanding multimodal interaction modalities (e.g., gestures, speech), and scaling the system across diverse platforms and complex workflows. Enhancing robustness in unstable environments and reducing computational costs will be key to broader adoption.
AI Executive Summary
In today’s rapidly evolving digital landscape, graphical user interfaces (GUIs) are central to user interaction, yet automating tasks within these interfaces remains a significant challenge. Traditional rule-based automation tools, such as Selenium and RPA platforms, depend heavily on fixed element identifiers and static scripts. These methods are brittle: slight interface changes can break automation, requiring manual repair. Recent advances in vision-based agents, powered by large language models (LLMs) and vision-language models (VLMs), have improved flexibility by operating directly over screenshots and natural language instructions. However, these systems often treat planning and adaptation as internal processes, limiting user visibility and control.
This paper introduces Plover, a novel plan-centric GUI automation system that externalizes task plans as persistent, inspectable, and editable artifacts. Unlike prior approaches, Plover emphasizes transparency and human oversight by providing explicit, visualized plans that users can review, modify, and repair during execution. The system architecture separates planning, execution, and repair components, with a shared workspace for plan visualization and multi-modal correction. The planner generates structured, versioned plans using heuristic search, while the executor grounds each step into pixel-based actions. An innovative intelligent replanning (IR) mechanism enables dynamic, localized plan adjustments triggered by user corrections or non-progress detection, ensuring task continuity.
Experimental results demonstrate Plover’s effectiveness: in a challenging benchmark of 38 tasks, initial autonomous execution failed in 26 cases. After applying collaborative repair, success rates increased to 88%, with an average of just over two interventions per task. The system effectively identifies failure modes such as spatial ambiguity and multi-step drift, and local plan edits significantly improve task completion. These findings highlight the importance of externalized, editable plans for robust GUI automation, especially in complex, high-stakes workflows.
Overall, this work advances GUI automation by making it more transparent, controllable, and adaptable. It opens avenues for integrating deep learning, richer multimodal interactions, and broader application across industries. Despite current limitations in highly unstable interfaces and performance in large-scale scenarios, Plover sets a foundational framework for future intelligent, human-in-the-loop automation systems.
Deep Analysis
Background
GUI自动化经历了从规则脚本到视觉感知驱动的转变。早期方法如Selenium和RPA工具依赖固定元素标识，易受界面变化影响。近年来，结合深度学习的视觉和自然语言模型（如GPT-4、VisualGPT）推动了基于像素的代理发展，支持更灵活的操作。然而，这些系统多为“黑箱”式，缺乏对计划的可视化和修正能力，限制了用户的理解和干预。自动化的透明性和修复性仍是核心挑战，尤其在复杂、多步骤任务中，界面结构频繁变化，修正难度大。
Core Problem
现有视觉GUI代理多为自主执行，缺乏对任务计划的可视化和修正机制，导致在界面变化或错误发生时，用户难以识别和修正偏差。系统的“黑箱”特性限制了用户对执行过程的理解和干预能力，尤其在长流程或高风险场景中，错误可能悄然累积，影响任务完成率。如何实现计划的外部化、可检视和局部修正，成为提升系统可靠性和用户信任的关键。
Innovation
本文提出将任务计划作为持久化、可编辑的外部工件，突破传统的内部推理限制，支持用户在执行过程中实时监控和修正。引入智能重规划（IR）机制，结合自然语言和截图标注，实现局部修正和连续修复，增强系统的透明度和可控性。系统架构将规划、执行和修正环节解耦，支持多模态交互，显著优于现有的自主代理和规则驱动方法，为GUI自动化带来全新的人机协作范式。
Methodology
- �� 规划器：基于启发式搜索和规则，生成结构化、版本化的任务计划，明确任务步骤和依赖关系。
- �� 执行器：利用像素匹配和动作映射，将计划中的步骤转化为鼠标键盘操作，实时观察界面状态。
- �� 交互界面：提供计划可视化、状态监控和多模态修正工具（自然语言、截图标注），支持用户监督和修正。
- �� 智能重规划（IR）：结合用户修正和非进展检测，动态修正未完成步骤，保持计划连续性。
- �� 多模态交互：支持自然语言指令和截图标注，增强空间修正的精确性。
- �� 反馈机制：系统实时显示执行状态，支持局部修正和计划版本管理。
Experiments
采用38个复杂GUI任务作为测试集，包括表单填写、数据迁移等，基线为自主执行模型。对比实验中，系统在修复失败任务后，成功率由不足到88%，平均修正次数为2.04。评估指标包括成功率、修正次数和用户满意度。还进行了场景模拟，验证计划外部化对任务连续性的支持。通过对失败类型的分析，识别空间模糊和偏差修正的关键场景，验证了系统的修复能力和交互效果。
Results
系统在挑战性任务中实现了显著修复效果，成功率由自动执行的不足到合作修复后大幅提升。局部修正策略有效应对空间模糊、多步骤偏差，提升了任务完成率。多模态交互增强了用户对计划的理解和干预能力，验证了计划外部化和可编辑机制的实用性。这些结果表明，计划的可视化和修正能力是提升GUI自动化修复性的重要因素。
Applications
该系统适用于复杂、长流程的企业自动化场景，如财务报表、数据录入、软件测试等。用户只需提供自然语言指令或截图标注，即可实现高效修正。未来，结合深度学习模型，可推广到多平台、多应用，提升自动化的智能化水平，减少人工干预成本。
Limitations & Outlook
系统在极端界面变化或复杂交互中仍存在计划偏离的风险，特别在界面结构高度不稳定时，修正效率降低。多模态交互依赖用户的准确操作，存在误差。高频修正场景下，性能和响应速度仍需优化，未来需结合深度学习提升鲁棒性和效率。
Plain Language Accessible to non-experts
想象你在厨房做饭，菜单和步骤就像一份食谱。传统自动化就像是用硬编码的指令，只能在菜单不变时用。一旦厨房布局变了，比如锅放错位置，指令就会失效。现在，Plover像是把食谱放在厨房的墙上，随时可以看、改，甚至用手指标记需要调整的地方。这样，当你发现锅放错了位置或需要加料时，可以直接在墙上的食谱上标记，系统就会根据新的指示调整操作。它让厨房变得更灵活，厨师（用户）可以随时干预，确保菜肴按预期完成。这就像把任务计划变成一份活的、可编辑的“厨房指南”，让自动化变得更聪明、更可靠。
ELI14 Explained like you're 14
想象你在学校里做科学实验，老师给你一份步骤说明。以前的自动化就像是照着说明机械操作，遇到问题就像锅坏了或材料不对，系统自己也不知道，不能帮你修正。现在，Plover就像是把实验步骤写在白板上，你可以用笔直接改写步骤，告诉系统哪里出错了。比如你发现某个步骤不对，可以用手指在白板上画圈，系统会根据你的标记调整下一步操作。这样，你就能更轻松地控制整个实验流程，不会因为一点点偏差就全乱了。它让自动化变得像你自己在操控实验一样，既灵活又可靠。
Glossary
Structured Plan (结构化计划)
一种明确表达任务步骤和依赖关系的计划，支持版本控制和编辑。技术上为版本化的树状结构，便于修正和追踪。
用于描述Plover中生成的任务计划，支持用户监督和修正。
智能重规划（IR）
结合用户修正和非进展检测的动态修正机制，保持计划连续性，局部调整未完成步骤。技术上为基于规则的局部重规划算法。
实现任务执行中的动态修正，确保任务连续性。
多模态交互
结合自然语言和视觉标注（截图、手势）实现用户指令和修正。技术上支持多模态输入融合与空间定位。
支持用户用自然语言或截图标记进行任务修正。
计划外部化
将任务计划作为持久化、可检视的外部工件存储，支持实时监控和编辑。技术上为计划的版本管理和可视化界面。
核心创新，增强系统的透明度和修正能力。
Open Questions Unanswered questions from this research
- 1 如何进一步提升系统在极端界面变化下的修复鲁棒性，特别是在界面结构高度不稳定或信息密集的场景中，仍是未来研究的重点。
Applications
Immediate Applications
企业自动化流程监控
利用Plover实现财务、HR等企业流程的自动化，用户可以在执行过程中实时监控和修正偏差，确保流程准确高效。
软件测试与界面调试
自动化测试脚本可通过计划外部化进行调试和修正，提高测试的灵活性和可靠性。
Long-term Vision
智能自主GUI代理
结合深度学习和多模态交互，未来实现更高水平的自主修复和适应能力，推动自动化向真正的智能代理转变。
Abstract
Graphical user interface (GUI) automation remains challenging in real-world environments, where dynamic layouts, unexpected dialogs, and evolving interface states can cause autonomous agents to drift from user intent. Recent vision-based multimodal agents improve flexibility by operating directly over screenshots and natural language instructions, but planning and adaptation often remain internal, limiting users' ability to inspect, supervise, or correct system behavior. We present Plover, a plan-centric vision-based GUI automation system that externalizes task plans and replanning as persistent, inspectable, and revisable artifacts. Through a planner--executor architecture, Plover supports explicit supervision of evolving execution, localized correction through editable plans, natural-language guidance, and screenshot-grounded interventions, while preserving prior progress during repair. A formative study with six participants informed the interaction design. We then evaluate Plover through benchmark failure-case repair and scenario-based workflow analyses. Our results show that many autonomous GUI-agent failures are structurally repairable when plans remain visible and interventions are localized, and that explicit replanning helps make GUI automation more transparent, controllable, and adaptable.