Rethinking Inference-Time Scaling in Local Computer-Use Agents: Failure Modes and Compute Tradeoffs
Study on inference-time scaling in local CUAs shows diminishing returns with increased computation.
Key Findings
Methodology
This paper systematically studies inference-time scaling in local computer-use agents across contextual, temporal, structural, and parallel dimensions. By evaluating Qwen3-VL-8B/30B-A3B, UI-TARS-1.5-7B, and OpenCUA-7B on the OSWorld benchmark, the impact of different scaling methods on task success and compute cost is analyzed.
Key Results
- Result 1: Contextual scaling improves trajectory stability and task accuracy, but gains saturate as token cost increases, shifting errors to premature false successes.
- Result 2: Temporal scaling reduces max-step stalls but does not significantly improve task success, indicating longer horizons often extend erroneous trajectories.
- Result 3: Structural decomposition introduces planning and formatting overhead in local two-stage agents, while parallel scaling partially mitigates these failures at substantial computational cost.
Significance
The study reveals the limitations of inference-time scaling in resource-constrained local models, emphasizing the importance of selective compute allocation and failure-aware control mechanisms. These findings are significant for improving the efficiency of local computer-use agents, especially in applications where privacy and cost efficiency are critical.
Technical Contribution
The technical contribution lies in revealing the non-linear dynamics of inference-time scaling in local models, showing that additional computation often changes failure modes rather than increasing task success. It proposes agentic frameworks designed around local model capabilities and limitations.
Novelty
This study is the first to systematically analyze the impact of inference-time scaling in local computer-use agents, providing a comprehensive understanding of contextual, temporal, structural, and parallel dimensions, filling a gap in existing research.
Limitations
- Limitation 1: The benefits of contextual scaling saturate as token cost increases, failing to significantly improve task success.
- Limitation 2: Temporal scaling fails to overcome the reasoning limitations of local models, leading to linear growth in operational costs.
Future Work
Future research can explore more efficient context management, failure-aware control mechanisms, and agentic framework designs that align with local model capabilities to further enhance the efficiency of local computer-use agents.
AI Executive Summary
With the rise of large language models, the paradigm of digital interaction has fundamentally shifted, enabling computer-use agents (CUAs) to autonomously navigate complex graphical user interfaces (GUIs). However, deploying these agents locally for enhanced privacy and cost efficiency remains challenging. This paper investigates the role of inference-time scaling in local CUAs, finding that additional computation often leads to diminishing returns and changes in failure modes.
The study evaluates models like Qwen3-VL, UI-TARS, and OpenCUA on the OSWorld benchmark, analyzing the impact of contextual, temporal, structural, and parallel scaling on task success and compute cost. Results show that contextual scaling improves trajectory stability and task accuracy, but gains saturate as token cost increases. Temporal scaling reduces max-step stalls but does not significantly improve task success, indicating longer horizons often extend erroneous trajectories.
Structural decomposition introduces planning and formatting overhead in local two-stage agents, while parallel scaling partially mitigates these failures at substantial computational cost. The study suggests that efficient local CUAs require selective compute allocation, failure-aware control mechanisms, and agentic frameworks designed around local model capabilities and limitations to enhance efficiency and task success rates.
Deep Analysis
Background
With the rapid development of large language models, computer-use agents (CUAs) have made significant progress in autonomously navigating complex graphical user interfaces (GUIs). These agents can perceive screen states and interaction histories to execute GUI actions and fulfill user instructions. However, deploying these agents locally for enhanced privacy and cost efficiency remains challenging, especially under hardware constraints.
Core Problem
The core problem is how to effectively scale inference-time in resource-constrained local environments to improve the performance of computer-use agents. While existing studies show that inference-time scaling can improve frontier agents through additional computation, its effectiveness for resource-constrained local models remains unclear.
Innovation
The core innovation of this paper lies in systematically studying inference-time scaling in local CUAs across contextual, temporal, structural, and parallel dimensions. By evaluating different models on the OSWorld benchmark, the study analyzes the impact of different scaling methods on task success and compute cost, revealing that additional computation often changes failure modes rather than increasing task success.
Methodology
- �� Contextual Scaling: Increase history length to improve trajectory stability.
- �� Temporal Scaling: Increase max steps to reduce max-step stalls.
- �� Structural Decomposition: Decouple reasoning into planning and execution stages.
- �� Parallel Scaling: Generate multiple candidate plans to improve plan selection.
Experiments
Experiments were conducted on the OSWorld benchmark, evaluating models like Qwen3-VL, UI-TARS, and OpenCUA. Metrics include task success rate, average steps, and prompt token usage. The study also analyzes the cost-accuracy tradeoffs of different scaling methods.
Results
Results show that contextual scaling improves trajectory stability and task accuracy, but gains saturate as token cost increases. Temporal scaling reduces max-step stalls but does not significantly improve task success. Structural decomposition introduces planning and formatting overhead, while parallel scaling partially mitigates these failures at substantial computational cost.
Applications
Application scenarios include local computer-use agents requiring high privacy and cost efficiency, such as personal assistants, automated office software, and smart home devices. These applications require efficient task execution under limited hardware resources.
Limitations & Outlook
Limitations include the saturation of contextual scaling benefits as token cost increases, the inability of temporal scaling to overcome local model reasoning limitations, and the planning and formatting overhead introduced by structural decomposition. Future research can explore more efficient context management and failure-aware control mechanisms.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. Contextual scaling is like remembering recipes you've made before so you don't repeat mistakes. Temporal scaling is like having more time to finish a complex dish, but if you don't know how to cook it, more time won't help. Structural decomposition is like splitting the cooking process into preparation and cooking stages, which might add extra steps and errors. Parallel scaling is like trying several different recipes at once to see which works best, but it requires more ingredients and time.
ELI14 Explained like you're 14
Hey there! Imagine you're playing a super complex game where you need to remember every step to win. Contextual scaling is like save points in the game, so you don't get stuck in the same place. Temporal scaling is like having more time to finish a level, but if you don't know how to play, more time won't help. Structural decomposition is like splitting the game into different levels, but it might get you lost. Parallel scaling is like trying different strategies at once, but it needs more energy and time.
Glossary
Inference-Time Scaling
A technique to improve performance by allocating additional computational resources during execution.
Used to enhance task success rates in computer-use agents.
Contextual Scaling
A method to improve trajectory stability by increasing history length.
Used to reduce repetitive loops and max-step stalls.
Temporal Scaling
A method to reduce max-step stalls by increasing max steps.
Provides more opportunities to reach a solution.
Structural Decomposition
A method to decouple reasoning into planning and execution stages.
Used to improve plan selection and reduce format errors.
Parallel Scaling
A method to improve plan selection by generating multiple candidate plans.
Partially mitigates failures introduced by structural decomposition.
Open Questions Unanswered questions from this research
- 1 How to improve local model reasoning capabilities without increasing compute cost?
- 2 How to effectively manage context to reduce token usage?
- 3 How to design agentic frameworks that align with local model capabilities?
Applications
Immediate Applications
Personal Assistants
Smart personal assistants running on local devices can perform complex tasks without compromising privacy.
Long-term Vision
Smart Homes
Implementing efficient task execution in smart home devices to enhance user experience and device interoperability.
Abstract
Deploying autonomous computer-use agents (CUAs) locally is increasingly important for privacy, cost efficiency, and practical usability, yet improving their performance under strict hardware constraints remains challenging. While recent studies show that inference-time scaling can improve frontier computer-use agents through additional computation during execution, its effectiveness for resource-constrained local models remains poorly understood. We present a systematic empirical study of inference-time scaling in local CUAs across contextual, temporal, structural, and parallel dimensions. We evaluate Qwen3-VL-8B/30B-A3B, UI-TARS-1.5-7B, and OpenCUA-7B on the OSWorld benchmark. Our results show that additional computation often yields diminishing returns while changing failure modes. Contextual scaling provides historical grounding that improves trajectory stability and task accuracy, but its gains saturate as token cost increases and failures shift from repetitive or stalled trajectories toward premature false successes. Temporal scaling similarly reduces max-step stalls, yet does not substantially improve task success, indicating that longer horizons often extend erroneous trajectories rather than correct them. We further find that structural decomposition can introduce planning and formatting overhead in local two-stage agents, while parallel scaling partially mitigates these failures at a substantial computational cost. Overall, our findings suggest that efficient local CUAs require selective compute allocation, failure-aware control mechanisms, and agentic frameworks designed around the capabilities and limitations of local models.