Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agents
Learning from Failure: Using failed trajectories for inference-time self-improvement, achieving a 6.6% success rate increase.
Key Findings
Methodology
This paper proposes a failure-driven self-improvement loop using multimodal large language models (MLLMs) to analyze failure modes, suggest inference-time solutions, and generate code patches to upgrade agents. This method achieves improvements without additional training costs through lightweight human verification.
Key Results
- Using the OpenCUA-72B model on the OSWorld benchmark, the success rate increased from 42.3% to 48.9%, an absolute improvement of 6.6 percentage points, with no additional training cost.
- The LLM analysis identified four major failure categories: grounding errors, competency gaps, knowledge deficiencies, and redundant loops.
- Generalization studies across different GUI benchmarks showed consistent improvements, especially for stronger models.
Significance
This research provides a novel data-driven self-improvement method by leveraging information from failed trajectories, complementing traditional success-driven pipelines, and significantly enhancing the efficiency and performance of computer-use agents.
Technical Contribution
The paper introduces a new failure-driven self-improvement loop, leveraging LLMs for failure analysis and inference-time improvements, offering a more efficient agent improvement mechanism compared to existing methods.
Novelty
This is the first study to transform failed trajectories into agent improvements, introducing a novel method for using LLMs for failure analysis and code patch generation at inference time.
Limitations
- The method relies on the LLM's analytical capabilities, which may underperform in complex failure scenarios.
- Requires some human verification, adding operational complexity.
Future Work
Future work could explore more automated verification processes to reduce human involvement and apply this method to a wider range of tasks and environments.
AI Executive Summary
Computer-use agents leverage multimodal large language models (MLLMs) to operate computers and complete tasks. Traditional methods rely on successful trajectories for self-improvement, overlooking the rich information in failed trajectories. This paper proposes a failure-driven self-improvement loop, using LLMs to analyze failure modes, suggest inference-time solutions, and generate code patches to upgrade agents. Experiments validate this method's effectiveness on the OSWorld benchmark, increasing the success rate from 42.3% to 48.9% without additional training costs. The study demonstrates that failure-driven self-improvement is a viable complement to success-driven pipelines, enabling more efficient agent enhancement. Future work will explore more automated verification processes and apply this method to a broader range of tasks and environments.
Deep Analysis
Background
Computer-use agents have gained significant attention for their utility and versatility, leveraging multimodal large language models (MLLMs) to observe computer states and predict actions. However, collecting large-scale, high-quality trajectories remains a major challenge.
Core Problem
Traditional self-improvement methods only utilize successful trajectories, ignoring the information in failed trajectories. These failures contain rich information about a model's weaknesses, making it a key problem to effectively leverage this information.
Innovation
This paper introduces a failure-driven self-improvement loop, using LLMs to analyze failure modes, propose inference-time solutions, and generate code patches. This innovation allows agents to self-improve at inference time without additional training costs.
Methodology
- �� Use LLMs to analyze failed trajectories and identify common failure modes.
- �� Propose inference-time solutions and generate code patches.
- �� Ensure the effectiveness of improvements through lightweight human verification.
- �� Validate improvements on the OSWorld benchmark.
Experiments
Experiments were conducted on the OSWorld benchmark using the OpenCUA-72B model. LLM analysis identified four major failure categories, and solutions were proposed for each. Results showed an increase in success rate from 42.3% to 48.9%.
Results
Results indicate that the failure-driven self-improvement method significantly enhances agent success rates, particularly in addressing grounding errors, competency gaps, knowledge deficiencies, and redundant loops.
Applications
This method can be directly applied to computer-use agents requiring efficient self-improvement, particularly in complex GUI environments.
Limitations & Outlook
The method relies on the LLM's analytical capabilities, which may underperform in complex failure scenarios. Additionally, lightweight human verification adds operational complexity.
Plain Language Accessible to non-experts
Imagine a factory where workers are on a production line. Traditional methods focus only on workers who produce quality products, ignoring those who produce defects. This method is like having a factory manager analyze these defects, identify issues, and provide improvement suggestions. This way, the overall efficiency of the factory is improved.
ELI14 Explained like you're 14
Imagine playing a complex video game where each time you fail, you get a hint about what went wrong. Then you can improve your strategy for the next attempt. This paper is about using those failure hints to make your game character stronger!
Glossary
Multimodal Large Language Model (MLLM)
A language model that combines multiple modalities (e.g., text, images) to understand and generate complex multimodal content.
Used for analyzing failed trajectories of computer-use agents.
Failure-driven Self-improvement
A method that improves system performance by analyzing failure cases.
The core method proposed in this paper.
Inference-time Solutions
Solutions proposed during system runtime to immediately improve system performance.
Generated by LLM after analyzing failure modes.
Code Patch
A small code modification used to fix or improve a software system.
Used to implement inference-time solutions.
OSWorld Benchmark
A standard testing environment for evaluating the performance of computer-use agents.
Used in experimental validation.
Open Questions Unanswered questions from this research
- 1 How can this method be applied to more complex tasks and environments?
- 2 How can human verification be reduced to increase automation?
Applications
Immediate Applications
Computer-use Agent Optimization
Improve agent performance in complex GUI environments using failure-driven self-improvement methods.
Long-term Vision
Intelligent System Self-improvement
Extend this method to a broader range of intelligent systems for more efficient self-improvement.
Abstract
Computer-use agents, which leverage multimodal large language models (MLLMs) to operate computers and complete tasks, have attracted significant attention for their utility and versatility. A major challenge in developing these agents is collecting large-scale, high-quality trajectories. The standard approach generates synthetic data through a self-improving loop: an agent is placed in a verifiable environment and iteratively fine-tuned on its successful trajectories. Despite its effectiveness, this paradigm exploits only successful trajectories and discards the failed ones, even though failures carry rich information about a model's weaknesses. In this work, we explore a complementary failure-driven self-improvement loop, a data-centric paradigm that turns failed trajectories into agent improvements. Specifically, we employ an LLM to diagnose failure modes, propose inference-time solutions, and generate code patches -- lightly verified by humans -- that upgrade the agent. We validate this approach with the state-of-the-art OpenCUA-72B model on the OSWorld benchmark, improving the success rate from 42.3% to 48.9%, a gain of 6.6 percentage points, without any additional training cost and with only modest inference overhead. Our results demonstrate that failure-driven self-improvement is a viable complement to success-based pipelines, enabling more efficient agent improvement.