RepoZero: Can LLMs Generate a Code Repository from Scratch?
RepoZero reformulates repository generation as behavior reproduction, using automated execution-based validation, revealing current LLM limitations in end-to-end code synthesis.
Key Findings
Methodology
This study introduces RepoZero, a benchmark transforming repo-level generation into a repository reproduction task. Models are given API specifications and tasked with re-implementing the entire repository so that its behavior matches the source. Validation relies on output equivalence via black-box testing, with cross-language constraints and sandbox environments to prevent data leakage. The ACE framework enables iterative test generation and error-driven refinement, significantly improving success rates. Experiments with multiple state-of-the-art LLMs show the best success rate is only 30%-55%, exposing the current gap in complex repo-level tasks.
Key Results
- In the Py2JS and C2Rust tasks, Claude-4.6-Sonnet achieved success rates of 55% and 54%, respectively. Overall, models averaged success rates between 30% and 55%. Incorporating ACE improved success by 15%-20%, demonstrating the effectiveness of iterative testing. Performance varied across difficulty levels and languages, indicating the complexity challenges models face.
- Results reveal that models can generate behaviorally consistent repositories, but often fall short in correctness and robustness. Output-based evaluation alone is insufficient; the automated execution validation ensures more reliable assessment.
- Comparison across models shows deficiencies in long-context retention and multi-module reasoning. Errors mainly stem from environment setup, dependency management, and complex interactions. ACE’s multi-round feedback loop mitigates some issues, boosting success rates.
Significance
This work addresses the critical challenge of objectively evaluating end-to-end repository generation. By establishing a fully automated, execution-based validation framework, it reduces subjective bias and enhances reproducibility. The benchmark exposes the current limitations of LLMs in complex software tasks, guiding future research toward more capable models. It bridges the gap from snippet-level code generation to full repository synthesis, with broad implications for automated software engineering and AI-driven development workflows.
Technical Contribution
RepoZero is the first scalable, verifiable benchmark for repository-level code generation, integrating API-based behavior matching, cross-language constraints, and sandbox environments. The ACE framework introduces multi-round test generation and error-driven refinement, significantly elevating success rates. This approach overcomes traditional patch-based evaluation limitations, enabling large-scale, automated assessment of complex code synthesis tasks. The methodology combines rigorous validation with practical scalability, advancing the state of AI-assisted software engineering.
Novelty
This is the first work to convert repository generation into a behavior reproduction problem validated by automated execution. It introduces cross-language synthesis constraints and a sandbox environment to prevent data leakage, ensuring genuine reasoning. The ACE iterative framework for test-driven refinement is a novel contribution, markedly improving success rates over previous benchmarks that relied on manual or heuristic evaluation. These innovations collectively push forward the frontier of autonomous code synthesis.
Limitations
- Despite improvements, success rates remain below industrial standards, especially in handling complex dependencies and logic. Long-context understanding and multi-module coordination are still weak points.
- Evaluation depends heavily on API and test case quality; poorly designed tests or incomplete API specifications can bias results.
- Experiments focus on specific languages and tasks; generalization to other languages or more complex scenarios requires further validation.
Future Work
Future directions include enhancing long-term memory and context management, integrating multimodal information (e.g., architecture diagrams), and expanding datasets across more languages and domains. Improving model reasoning capabilities and reducing reliance on handcrafted tests will be key. Additionally, combining this framework with continuous integration pipelines could enable real-world automated software development, making AI-assisted coding more practical and reliable.
AI Executive Summary
Automated code generation has rapidly advanced with large language models like GPT and Codex, yet evaluating their ability to generate entire software repositories remains a significant challenge. Existing benchmarks primarily focus on patch editing or rely on human judgment, which introduces bias and limits scalability. To address this, we introduce RepoZero, a novel benchmark that reformulates repository-level generation as a behavior reproduction task. Given only API specifications, models are tasked with re-implementing the entire repository so that its behavior matches the source, validated through automated execution-based testing.
This approach ensures a fully objective, scalable, and reproducible evaluation. By leveraging black-box testing, cross-language constraints, and sandbox environments, RepoZero effectively mitigates data leakage and shortcut solutions. To further improve success rates, we propose the Agentic Code-Test Evolution (ACE) framework, which iteratively generates, tests, and refines code based on execution feedback. Extensive experiments with multiple state-of-the-art LLMs reveal that even the best models only achieve success rates of 30%-55%, exposing substantial gaps in current capabilities.
The significance of this work lies in establishing a rigorous, automated evaluation paradigm for complex software synthesis tasks. It pushes the frontier from snippet-level code generation to full repository construction, with broad implications for AI-driven software engineering. Future research will focus on enhancing long-term memory, reasoning, and cross-modal understanding, aiming to close the gap between current models and industrial-grade automation.
Deep Dive
Abstract
Large Language Models (LLMs) have recently shown remarkable progress in code generation, yet their ability to construct complete software repositories from scratch remains poorly understood. A fundamental bottleneck is the lack of verifiable and scalable evaluation: existing benchmarks either focus on patch-based editing or rely on human or LLM-based judgments, which introduce bias and limit reproducibility. In this work, we present RepoZero, the first benchmark that enables fully automated, execution-based verification of repository-level generation from scratch. Our key idea is to reformulate generation as repository reproduction: given only API specifications, an agent must re-implement an entire repository such that its behavior matches the original implementation. This design allows for strict black-box validation via output equivalence, while naturally supporting large-scale construction by reusing existing open-source repositories. To further mitigate data leakage and shortcut solutions, we introduce cross-language constraints and a sandboxed evaluation protocol. Building on this benchmark, we propose an Agentic Code-Test Evolution (ACE) framework that performs iterative test generation and error-driven refinement, enabling effective test-time scaling for repository-level synthesis. Extensive experiments across multiple state-of-the-art LLMs and agent frameworks reveal that even the strongest LLM agents achieve only limited pass rates (30\% - 55\%), exposing a substantial gap between current capabilities and real-world software development requirements. Our results establish RepoZero as a challenging, scalable, and reliable testbed for end-to-end code generation, and highlight self-verification via test generation as a critical direction for advancing LLM-based coding agents.