StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents
StateAct leverages program state for long-horizon tasks, achieving 26.9% success, ninefold cost reduction, outperforming pixel-based methods.
Key Findings
Methodology
StateAct employs a code-centric multi-agent framework, where the main agent directly manipulates persistent program state via files, DOM, and databases. It integrates a state-first action space, an independent verification gate, and context management to sustain long-term task execution. Visual interaction is delegated to a dedicated GUI subagent, invoked only for visual subgoals, minimizing visual reliance. The system's core innovation lies in grounding actions, verification, and memory in program state, which reduces error accumulation over hundreds of steps. Experiments with Claude Opus 4.8 demonstrate a success rate of 26.9%, a ninefold reduction in per-task cost, validating the approach’s efficiency and robustness.
Key Results
- On OSWorld 2.0, StateAct increased binary success from 20.6% to 26.9%, partial success from 54.8% to 61.6%, surpassing all public entries. Cost per task dropped from ~$72 to ~$7.8, nearly nine times cheaper. Ablation studies confirmed that removing state operations drops success by over 10%, emphasizing their importance. The architecture’s combination of state manipulation, independent verification, and context management significantly improved task continuity and reliability, setting new benchmarks for long-horizon automation.
- Compared to traditional visual-only methods, StateAct’s success rate improved by 6.3 percentage points, with a corresponding cost reduction. Its verification gate effectively detects structural errors, though value reasoning remains a challenge. The system’s design demonstrates that grounding in program state enhances both efficiency and accuracy, especially over extended task sequences.
- Overall, the results highlight that state-grounded agents can outperform pixel-based counterparts in complex, multi-step tasks, with substantial gains in success rate, cost, and robustness. This validates the core hypothesis that direct program state access is crucial for scalable, reliable automation.
Significance
This work fundamentally shifts the paradigm from perception-centric to state-centric automation, addressing core limitations of pixel-based models such as information loss and error accumulation. By grounding actions, verification, and memory in program state, it enables more reliable, scalable, and interpretable long-term autonomous agents. The architecture’s modularity and robustness open pathways for deploying AI in real-world enterprise automation, software testing, and data management, where correctness and efficiency are paramount. The approach also offers a blueprint for future research integrating deep reasoning with programmatic state manipulation, promising more intelligent and trustworthy autonomous systems.
Technical Contribution
The paper introduces a novel multi-agent system that centers around program state, contrasting with existing perception-driven models. Key innovations include a state-first action space, an independent verification gate grounded in state, and context management techniques that preserve long-term task continuity. The architecture supports flexible delegation, minimal visual reliance, and robust error detection. Theoretical contributions include formalizing the benefits of state-grounding in reducing error drift and enhancing verification. Engineering-wise, it demonstrates scalable implementation across diverse tasks, validated by extensive experiments, establishing a new standard for long-horizon autonomous agents.
Novelty
This is the first comprehensive system to treat program state as the primary interface for long-horizon automation, moving beyond pixel-based perception. Its core innovation lies in integrating state manipulation, independent structural verification, and context management into a unified multi-agent framework. Unlike prior works focusing solely on visual perception or API-based actions, StateAct combines both, emphasizing the importance of internal program state for reliable, scalable automation. This approach fundamentally redefines how autonomous agents can operate over extended sequences, addressing longstanding issues of information drift and structural errors.
Limitations
- The verification mechanism primarily detects structural errors, with limited capacity for deep value reasoning, leaving some logical or inferential errors unaddressed.
- Performance in highly unstructured or non-programmatic environments remains limited, requiring further adaptation for broader applicability.
- Computational overhead from maintaining multiple agents and context management can be significant, necessitating optimization for real-time deployment.
Future Work
Future research will focus on integrating deep reasoning and value-based verification to improve error detection beyond structure. Enhancing the recursive capabilities of subagents and expanding their autonomy could enable even longer task horizons. Additionally, optimizing system efficiency and scalability will be critical for industrial applications. Exploring learning-based verification and adaptive context management may further improve robustness and generalization, paving the way for autonomous systems capable of complex, real-world tasks over extended durations.
AI Executive Summary
Traditional AI agents for desktop automation rely heavily on visual perception, which is inherently lossy and prone to errors over long sequences. These methods often struggle with maintaining task continuity, detecting structural failures, and ensuring correctness in complex workflows. To address these issues, StateAct introduces一种以程序状态为核心的多智能体架构，强调在操作和验证中直接利用程序内部的真实状态信息。该系统由三部分组成：主代理负责通过代码操作程序状态，子代理专门处理视觉交互，验证门独立核查任务完成情况。核心创新在于将动作空间、验证机制和上下文管理融合，显著提升长时任务的成功率和效率。实验结果显示，基于Claude Opus 4.8模型，成功率达26.9%，成本降低九倍，优于所有公开系统。该架构突破了以像素为中心的感知限制，为自动化代理的未来发展提供了新方向。尽管如此，系统在深层推理和非结构化环境中仍存在局限，未来将结合深度学习和学习验证机制，持续优化性能，推动行业应用落地。
Deep Dive
Abstract
Computer-use agents are usually improved by strengthening perception: better models for reading a screenshot and choosing where to click. Yet a screenshot is only a lossy rendering of the underlying program state, e.g., the files, application backends, and DOM that hold the task data. Different states can produce the same pixels, while code can inspect and modify that state directly. StateAct is a code-first, multi-agent harness built around this distinction. Its main agent works directly with program state by using code, while a dedicated GUI subagent handles screenshot-and-click interaction on the few subgoals that need it, just 28 of 108 tasks and 1.1% of main-agent steps. The same direct access to program state also supports verification: an independent finish gate double-checks the saved result for structural failures, e.g., output that is missing, unsaved, or written to the wrong path. To stay on track over hundreds of steps, the main agent hands subgoals to fresh subagents, keeping its own context focused. On OSWorld 2.0, StateAct lifts Claude Opus 4.8 from 20.6% to 26.9% on binary success, and from 54.8% to 61.6% on partial success, at ~ 9x lower cost per task than the same model driven by screenshots alone; a code-only variant with no GUI subagent reaches only 45.9% partial, below that screenshot-based baseline's 54.8%. In general, grounding action, verification, and memory in state, what we call state-grounding, shifts the main bottleneck from perception toward reasoning: failures depend more on what the agent thinks than on what it sees.