NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents
NL2Repo-Bench evaluates long-horizon repository generation of coding agents, with average pass rates below 40%, highlighting significant challenges.
Key Findings
Methodology
This study introduces NL2Repo-Bench, a benchmark designed to assess the ability of coding agents to autonomously generate complete software repositories from a single natural language requirements document. The process involves selecting representative open-source repositories based on complexity, maturity, and testability. Each target repository undergoes reverse engineering via AST analysis to extract API details, which inform the construction of detailed, structured task specifications encompassing project description, support materials, API usage guides, and implementation nodes. Environment setup employs Docker containers with dependency pinning, ensuring deterministic execution. The evaluation strictly measures functional correctness by executing the generated code against the original pytest test suite within isolated environments. Multiple state-of-the-art models, including DeepSeek, Claude, and GPT-5, are evaluated across 104 diverse tasks. Results reveal that even the best models achieve less than 40% average test pass rate, with common failure modes such as premature termination, loss of global coherence, fragile dependency handling, and inadequate long-term planning.
Key Results
- Across 104 tasks, the top-performing models achieve an average test pass rate below 40%, with full repository reconstruction rarely successful, indicating that long-horizon repository generation remains an open challenge.
- Performance varies significantly across difficulty levels, with easy tasks exceeding 50% success, while hard tasks fall below 20%, emphasizing the complexity of maintaining global coherence over extended sequences.
- Failure analysis uncovers systematic issues such as early termination due to overconfidence, loss of architectural consistency, dependency fragility, and difficulties in executing and verifying plans over hundreds of interaction steps.
Significance
This work establishes a rigorous, objective benchmark for evaluating the long-term reasoning and planning capabilities of autonomous coding agents. It addresses a critical gap in current AI research, which predominantly focuses on short-term code synthesis or repair tasks. By emphasizing end-to-end repository construction driven solely by natural language, NL2Repo-Bench pushes the boundary of what AI systems can achieve in automated software engineering. The benchmark's strict execution-based evaluation ensures that progress is measured by real software functionality, not proxy metrics or subjective judgments. This has profound implications for advancing AI towards true artificial general intelligence in the domain of software development, with potential industry impacts including faster deployment cycles, reduced manual effort, and more reliable automation pipelines.
Technical Contribution
The primary technical contribution lies in designing an end-to-end, verifiable benchmark that challenges models to generate complete, installable Python repositories from scratch. The framework integrates AST-based API extraction, detailed specification construction, and Docker-based deterministic environments. It introduces a systematic failure mode analysis, revealing core bottlenecks such as global architectural coherence, dependency fragility, and long-term planning. The benchmark supports multi-model evaluation, providing insights into the current state-of-the-art capabilities and limitations. This work also establishes a comprehensive dataset of 104 tasks, covering diverse application domains and complexity levels, serving as a valuable resource for future research in long-horizon AI reasoning and autonomous software engineering.
Novelty
NL2Repo-Bench is the first benchmark explicitly targeting the challenge of from-scratch, long-horizon repository generation solely based on natural language specifications. Unlike prior benchmarks that focus on code completion, repair, or partial reconstruction, this work emphasizes the entire software lifecycle, including architecture design, dependency management, and multi-file implementation. Its strict execution-based evaluation ensures that generated repositories are functionally correct and fully installable, setting a new standard for objective assessment. The integration of AST analysis for API extraction and Docker environments for reproducibility further distinguishes this benchmark from existing efforts, making it a pioneering step toward autonomous, end-to-end software construction.
Limitations
- Models tend to prematurely terminate interactions due to overconfidence, limiting their ability to sustain long-term planning and execution over hundreds of steps.
- Dependency and cross-file dependency management remain fragile, often leading to broken builds or inconsistent architectures, especially in complex projects.
- The current evaluation framework primarily measures functional correctness via pytest, but does not yet assess code quality, maintainability, or scalability, which are crucial for industrial adoption.
Future Work
Future directions include developing models with improved global planning and self-verification capabilities, enabling sustained reasoning over extended sequences. Incorporating knowledge graphs or multimodal data could enhance dependency handling and architectural coherence. Expanding the benchmark to include more diverse application domains and larger-scale projects will better reflect real-world scenarios. Additionally, integrating qualitative assessments of code quality and maintainability will provide a more holistic evaluation. Ultimately, the goal is to realize autonomous agents capable of constructing complex, reliable, and scalable software systems from minimal input, significantly accelerating software development cycles and reducing manual effort.
AI Executive Summary
The rapid evolution of large language models (LLMs) has transformed the landscape of automated software development. Early tools like GPT-3 and Codex demonstrated remarkable capabilities in generating isolated functions or repairing code snippets. However, these systems primarily excelled at short-term, localized tasks, leaving the challenge of constructing entire software repositories from scratch largely unaddressed. As AI research pushes toward autonomous software engineering, a critical gap has emerged: how to evaluate an agentβs ability to perform long-horizon, system-level tasks that involve architecture design, dependency management, multi-file coordination, and iterative debugging.
NL2Repo-Bench addresses this gap by establishing a rigorous, execution-based benchmark for assessing the long-term reasoning and planning skills of coding agents. The benchmark draws from real-world open-source Python libraries, carefully selected to ensure complexity, maturity, and verifiability. Each task begins with a single natural language requirements document, devoid of any scaffolding or pre-existing code. The agent must interpret this document, reverse engineer the underlying architecture, manage dependencies, implement multi-module logic, and produce a fully installable package. This process mimics real-world software development, where architects and engineers must reason globally and persistently over extended periods.
The evaluation framework employs Docker containers to guarantee environment consistency and executes the generated repositories against the original pytest test suites. This strict, objective measure of correctness ensures that the generated code functions as intended, rather than relying on subjective judgments or proxy metrics. The experiments involve multiple state-of-the-art models, including DeepSeek, Claude, and GPT-5, evaluated across 104 diverse tasks. Results reveal a sobering reality: even the most advanced models achieve an average test pass rate below 40%, with full repository reconstruction rarely successful.
Deeper analysis uncovers fundamental failure modes, such as premature termination of interactions, loss of global architectural coherence, fragile dependency handling, and inability to maintain plans over hundreds of steps. These findings highlight that current models lack the necessary long-term reasoning, planning, and verification mechanisms to succeed in complex, end-to-end software construction. The work underscores the urgent need for research into models capable of sustained, global reasoning, self-verification, and robust dependency management.
NL2Repo-Bench provides both a diagnostic tool and a research target, guiding future efforts toward truly autonomous, long-horizon software agents. Its comprehensive dataset, rigorous evaluation protocol, and insightful failure analysis set a new standard for benchmarking AI in software engineering. As models improve, this benchmark will serve as a critical yardstick for measuring progress toward AI systems that can independently build complex, reliable software systems from minimal instructions, revolutionizing the future of software development.
Deep Dive
Abstract
Recent advances in coding agents suggest rapid progress toward autonomous software development, yet existing benchmarks fail to rigorously evaluate the long-horizon capabilities required to build complete software systems. Most prior evaluations focus on localized code generation, scaffolded completion, or short-term repair tasks, leaving open the question of whether agents can sustain coherent reasoning, planning, and execution over the extended horizons demanded by real-world repository construction. To address this gap, we present NL2Repo Bench, a benchmark explicitly designed to evaluate the long-horizon repository generation ability of coding agents. Given only a single natural-language requirements document and an empty workspace, agents must autonomously design the architecture, manage dependencies, implement multi-module logic, and produce a fully installable Python library. Our experiments across state-of-the-art open- and closed-source models reveal that long-horizon repository generation remains largely unsolved: even the strongest agents achieve below 40% average test pass rates and rarely complete an entire repository correctly. Detailed analysis uncovers fundamental long-horizon failure modes, including premature termination, loss of global coherence, fragile cross-file dependencies, and inadequate planning over hundreds of interaction steps. NL2Repo Bench establishes a rigorous, verifiable testbed for measuring sustained agentic competence and highlights long-horizon reasoning as a central bottleneck for the next generation of autonomous coding agents.
References (20)
OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models
Siming Huang, Tianhao Cheng, J. Liu et al.
From Code Foundation Models to Agents and Applications: A Practical Guide to Code Intelligence
Jian Yang, Wei Zhang, Shark Liu et al.
System-on-card
C. Nieland
Cloud
M. Harrison
Evaluating Large Language Models Trained on Code
Mark Chen, Jerry Tworek, Heewoo Jun et al.
Program Synthesis with Large Language Models
Jacob Austin, Augustus Odena, Maxwell Nye et al.
RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems
Tianyang Liu, Canwen Xu, Julian McAuley
ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation
Xueying Du, Mingwei Liu, Kaixin Wang et al.
Code Llama: Open Foundation Models for Code
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle et al.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Carlos E. Jimenez, John Yang, Alexander Wettig et al.
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
John Yang, Carlos E. Jimenez, Alexander Wettig et al.
DDK: Distilling Domain Knowledge for Efficient Large Language Models
Jiaheng Liu, Chenchen Zhang, Jinyang Guo et al.
OpenHands: An Open Platform for AI Software Developers as Generalist Agents
Xingyao Wang, Boxuan Li, Yufan Song et al.
M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluation
Jiaheng Liu, Ken Deng, Congnan Liu et al.
DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
DeepSeek-AI, A. Liu, Aoxue Mei et al.
Commit0: Library Generation from Scratch
Wenting Zhao, Nan Jiang, C. Lee et al.
SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?
Samuel Miserendino, Michele Wang, Tejal Patwardhan et al.
CodeCriticBench: A Holistic Code Critique Benchmark for Large Language Models
Alex L. Zhang, M. Dong, Jiaheng Liu et al.
PaperBench: Evaluating AI's Ability to Replicate AI Research
Giulio Starace, Oliver Jaffe, Dane Sherburn et al.
Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving
Daoguang Zan, Zhirong Huang, Wei Liu et al.
Cited By (20)
DeNovoSWE: Scaling Long-Horizon Environments for Generating Entire Repositories from Scratch
CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation
Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios
SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades
SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades
SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cycle
RepoZero: Can LLMs Generate a Code Repository from Scratch?
Constraint Decay: The Fragility of LLM Agents in Backend Code Generation
ProgramBench: Can Language Models Rebuild Programs From Scratch?
"What Did It Actually Do?": Understanding Risk Awareness and Traceability for Computer-Use Agents
SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents
SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution
Specification-Driven Generation and Evaluation of Discrete-Event World Models via the DEVS Formalism
BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?
SWE-Hub: A Unified Production System for Scalable, Executable Software Engineering Tasks
Discordance Between Textual Reasoning and Visual Interpretation in Large Language Models for Low Back Pain: Cross-Sectional Quantitative Evaluation and Exploratory Multimodal Stress Test
Immersion in the GitHub Universe: Scaling Coding Agents to Mastery
ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development
RepoGenesis: Benchmarking End-to-End Microservice Generation from Readme to Repository