ANCHOR: Branch-Point Data Generation for GUI Agents

Recent advances in multimodal models have significantly improved GUI interaction capabilities, enabling agents to perform complex desktop tasks. Prior works like UI-TARS, Agent S, and others have demonstrated progress but are limited by reliance on manual data collection or short synthetic trajectories. Data scarcity remains a bottleneck, especially for long-horizon, multi-step workflows. Synthetic data generation approaches, including goal-conditioned and exploration-driven methods, have attempted to address this but often produce low-signal, short, or noisy trajectories. Consequently, models trained on such data struggle with generalization and robustness. The need for scalable, high-quality, diverse datasets is urgent to push the field forward, especially for real-world applications involving complex multi-step procedures across various desktop environments.

The core challenge is generating large-scale, diverse, and high-fidelity GUI interaction data efficiently. Manual annotation is prohibitively expensive and not scalable, while existing synthetic methods lack the ability to produce long, task-relevant, and coherent trajectories. Goal-conditioned synthesis often suffers from bias towards simpler tasks, and exploration-based approaches generate many low-signal wandering trajectories. This limits the training data's usefulness for robust model fine-tuning, especially across different applications and operating systems. Addressing this bottleneck requires a method that can systematically expand verified demonstrations into diverse, high-quality trajectories with minimal manual effort.

ANCHOR introduces a structured trajectory expansion framework based on UI state change detection. Key innovations include: 1) identifying branch points via GPT-5.1, which detects meaningful UI changes; 2) generating multiple task variants conditioned on current GUI states, ensuring task diversity; 3) employing a verifier to automatically confirm task success, reducing manual oversight; 4) implementing step-level filtering and denoising to improve supervision quality. Unlike prior methods relying solely on goal-conditioned prompts or exploration, ANCHOR systematically explores multiple branches around verified seeds, producing longer, task-relevant trajectories. This approach enhances data diversity, quality, and scalability, enabling more effective training of GUI agents.

• �� Collect high-quality seed trajectories from human validation of successful runs in OSWorld and WindowsAgentArena.
• �� Use GPT-5.1 to analyze each seed trajectory and identify decision points where UI state changes significantly.
• �� At each branch point, generate concise summaries of progress and propose new task instructions grounded in current GUI state.
• �� Replay environment to branch point state, then execute proposed tasks with a GUI agent, refining instructions dynamically based on environment feedback.
• �� Summarize completed trajectories into high-level task descriptions, abstracting low-level actions.
• �� Apply a verifier to confirm task completion, filtering out unsuccessful or incoherent trajectories.
• �� Implement step-level reasoning conditioned on downstream tasks, using GPT-5.1 to validate actions against visual changes.
• �� Post-branch, run intention-consistency checks to remove noisy or irrelevant steps, ensuring supervision signal purity.

The dataset comprises 1777 trajectories, with 1174 from Ubuntu and 603 from Windows, averaging 17.24 steps per trajectory at a cost of $0.47 each. Models like Claude Sonnet 4.5, GPT-5.1, and Qwen3-VL-8B are employed for execution, proposal, and filtering. Evaluation involves success rate comparisons against zero-shot, goal-conditioned, and human datasets across OSWorld and WindowsArena benchmarks. Ablation studies demonstrate the importance of branch point detection, filtering, and denoising. Fine-tuning models on ANCHOR data consistently improves success rates, confirming the method's effectiveness in producing high-quality, diverse trajectories suitable for training robust GUI agents.

ANCHOR-generated data boosted Qwen3-VL-8B success rate from 16.82% to 20.56% on OSWorld, and similar improvements were observed across other models and benchmarks. Success rates on WindowsArena increased by approximately 6%, outperforming traditional synthetic and human datasets. Ablation experiments confirmed that branch point identification, filtering, and denoising are critical for performance gains. Longer, task-oriented trajectories contributed to better model generalization, especially in complex multi-step workflows. These results validate the effectiveness of structured, state-grounded data synthesis for GUI tasks.

This approach can be applied to automate complex desktop workflows, software testing, and enterprise automation, where large-scale, high-quality interaction data is essential. It enables rapid dataset expansion with minimal manual effort, facilitating training of more capable, generalizable GUI agents. The method's scalability and robustness make it suitable for diverse applications across industries, including finance, healthcare, and IT management. Future integration with reinforcement learning and multi-modal inputs could further enhance autonomous desktop intelligence.

Dependence on high-quality seed demonstrations may limit scalability if initial data quality is poor. Identifying meaningful branch points in highly dynamic or cluttered interfaces remains challenging, potentially affecting trajectory diversity. Computational costs are high due to multi-model collaboration and extensive filtering, requiring optimization for large-scale deployment. Additionally, the method's effectiveness in highly complex or rapidly changing environments needs further validation. Future work should focus on automating seed selection, improving branch detection, and reducing computational overhead.