RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents
RecreationWorld provides scalable, verifiable environments for hybrid agents, with GPT-6 Astra scoring 58.1% on RecreationBench.
Key Findings
Methodology
RecreationWorld framework offers reproducible environments across Ubuntu, macOS, Windows, Android, and Web, allowing agents to explore, implement, and verify software without a prescribed workflow. It uses high-quality open-source applications to generate training trajectories, supporting cross-platform GUI control and coding tools.
Key Results
- GPT-6 Astra scored 58.1% overall on RecreationBench, passing all programmatic tests on only 2.8% of tasks.
- Agents reproduced static interface structures more reliably than interactions and computed outputs.
- Generated applications were smaller and more monolithic than their references.
Significance
RecreationWorld provides a scalable and verifiable environment for hybrid computer-use agents, addressing the lack of interaction between GUI and command-line operations in traditional agents. This framework facilitates self-improvement and capability transfer across platforms and tasks.
Technical Contribution
RecreationWorld significantly expands the operational space of hybrid agents by providing a unified cross-platform environment, enabling seamless switching in real digital workspaces. Its verification mechanism is based on executable behavior rather than source-code similarity, offering new engineering possibilities.
Novelty
RecreationWorld is the first to integrate hybrid agent operations and coding capabilities in a long-horizon process, providing a self-verifying loop that surpasses traditional single-mode interactions.
Limitations
- Agents performed poorly in reproducing interactions and computed outputs, with generated applications being smaller and more monolithic.
- Passed all programmatic tests on only 2.8% of tasks.
Future Work
Future work can focus on improving agent performance in complex interactions and computed outputs, and optimizing the structure and size of generated applications.
AI Executive Summary
RecreationWorld framework provides scalable and verifiable environments for hybrid computer-use agents, addressing the lack of interaction between GUI and command-line operations in traditional agents. By offering reproducible environments across Ubuntu, macOS, Windows, Android, and Web, the framework allows agents to explore, implement, and verify software without a prescribed workflow.
In experiments, GPT-6 Astra achieved an overall score of 58.1% on RecreationBench, passing all programmatic tests on only 2.8% of tasks. Agents reproduced static interface structures more reliably than interactions and computed outputs, while generated applications were smaller and more monolithic than their references.
RecreationWorld's verification mechanism is based on executable behavior rather than source-code similarity, offering new engineering possibilities. Future work can focus on improving agent performance in complex interactions and computed outputs, and optimizing the structure and size of generated applications.
Deep Analysis
Background
As computer-use agents advance, traditional agents lack interaction between GUI and command-line operations. RecreationWorld framework addresses this by providing a cross-platform verifiable environment, allowing agents to explore, implement, and verify software without a prescribed workflow.
Core Problem
Traditional agents lack interaction between GUI and command-line operations, leading to poor performance in real digital work. A framework that integrates both is needed to support agent self-improvement across platforms and tasks.
Innovation
RecreationWorld provides a unified cross-platform environment, allowing agents to explore, implement, and verify software without a prescribed workflow. Its verification mechanism is based on executable behavior rather than source-code similarity.
Methodology
- �� Provide reproducible cross-platform environments
- �� Use high-quality open-source applications for training trajectories
- �� Support cross-platform GUI control and coding tools
- �� Verification mechanism based on executable behavior
Experiments
Tested 250 tasks on RecreationBench, covering different domains and platforms. Used GPT-6 Astra to evaluate agent performance across platforms and tasks.
Results
GPT-6 Astra scored 58.1% overall on RecreationBench, passing all programmatic tests on only 2.8% of tasks. Agents reproduced static interface structures more reliably than interactions and computed outputs.
Applications
RecreationWorld can be used for developing and testing hybrid computer-use agents, supporting self-improvement and capability transfer across platforms and tasks.
Limitations & Outlook
Agents performed poorly in reproducing interactions and computed outputs, with generated applications being smaller and more monolithic. Future work can focus on optimizing the structure and size of generated applications.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. RecreationWorld is like a smart chef assistant that helps you seamlessly switch between different kitchen appliances, like the oven and blender. It not only helps you find recipes but also adjusts ingredient proportions to ensure every dish is perfect. This assistant works in various kitchen environments, such as home kitchens, restaurant kitchens, and even outdoor barbecues.
ELI14 Explained like you're 14
Hey there! Imagine you're playing a super complex game where you need to control multiple characters at once. RecreationWorld is like a super smart game assistant that helps you switch between different scenes, like forests and castles. It not only helps you find hidden treasures but also helps you defeat big bosses! This assistant works in different game environments, like computer games, mobile games, and even web games.
Glossary
Hybrid Agent
An intelligent agent capable of seamlessly switching between GUI and command-line operations.
Core component in the RecreationWorld framework.
Verifiable Environment
An environment that allows for the verification of agent behavior.
Cross-platform environment provided by RecreationWorld.
RecreationBench
A benchmark test set for evaluating hybrid agent performance.
Test set containing 250 tasks.
GPT-6 Astra
The model that performed best on RecreationBench.
Achieved an overall score of 58.1% in experiments.
Open-source Application
Applications that are freely available for use and modification.
High-quality applications used for generating training trajectories.
Open Questions Unanswered questions from this research
- 1 How to improve agent performance in complex interactions and computed outputs?
- 2 How to optimize the structure and size of generated applications?
- 3 How to increase the pass rate of programmatic tests?
Applications
Immediate Applications
Development Testing
Developers can use RecreationWorld to test hybrid agents' performance across different platforms, optimizing their capabilities.
Long-term Vision
Intelligent Assistant
RecreationWorld can evolve into an intelligent assistant, helping users seamlessly switch across different digital environments.
Abstract
Computer-use agents (CUAs) have advanced along two separate lines: graphical interaction and software development through code and the command line. Real digital work requires both, interleaved rather than stacked end to end. We study hybrid CUAs that autonomously decide when to explore an interface, implement software, and run and visually verify their artifacts. We introduce RecreationWorld, a five-platform framework built around recreation: given a running reference, an agent must discover its behavior and build a faithful implementation with no prescribed workflow. RecreationWorld provides reproducible environments on Ubuntu, macOS, Windows, Android, and Web, plus a unified harness with native GUI control and coding tools. The running reference serves as an oracle for hidden behavioral tests, providing execution-grounded rewards. We scale trajectory generation with high-quality open-source applications. Models trained on these trajectories improve across five out-of-distribution coding and hybrid computer-use benchmarks and more frequently verify their rendered outputs, providing evidence of transfer beyond recreation. For held-out evaluation, we introduce RecreationBench, comprising 250 diverse tasks across domains and platforms. Reference-grounded programmatic and visual assertions cover action-conditioned outcomes at multiple interaction depths; each is validated on the reference and by human reviewers before the suite is frozen for automatic scoring. GPT-6 Astra leads at 58.1% overall, but passes all programmatic tests on just 2.8% of tasks. Agents reproduce static interface structure more reliably than interactions and computed outputs, while generated applications remain smaller and more monolithic than their references. We release the benchmark, environments, and test suites.