ProgramBench: Can Language Models Rebuild Programs From Scratch?
ProgramBench evaluates LMs' ability to rebuild programs from scratch; the best model passes 95% of tests on only 3% of tasks, spanning CLI tools to complex software.
Key Findings
Methodology
This study introduces ProgramBench, a benchmark where models are given only a binary executable and documentation, tasked with designing architecture and implementing code to match behavior. Using automated fuzzing, behavioral tests are generated to verify functional equivalence without exposing implementation details. Nine recent language models, including Claude and GPT variants, are evaluated on 200 diverse tasks from open-source repositories, covering small CLI utilities to large systems like FFmpeg and PHP. Models must autonomously decide on architecture, modularization, and abstractions, reflecting real-world software development challenges. Metrics include task resolution and test pass rates, revealing significant gaps in current model capabilities.
Key Results
- All models failed to fully resolve any task; the highest achieved only 3% of tasks with over 95% test pass rate. Models prefer monolithic, single-file solutions, diverging from human modular design. Claude Opus 4.7 performed best on some tasks, passing 95% of tests, but overall success remains limited.
- Complex software such as FFmpeg and PHP interpreters remain out of reach, while simpler CLI tools like fzf and jq show better performance. Generated code tends to be longer, less modular, and less aligned with human design principles. Behavior validation via automated test suites confirms functional correctness but highlights architectural deficiencies.
- Test suites generated by the models achieve comparable coverage to native developer tests, validating behavior consistency. However, models' architectural choices remain suboptimal, indicating a need for improved abstraction and modularization capabilities.
Significance
This work pioneers a comprehensive, behavior-based evaluation of models' ability to reconstruct entire software systems, addressing a critical gap in assessing AI's role in software engineering. By focusing on executable behavior rather than source code structure, it offers a more realistic measure of a model's design and implementation skills. The benchmark exposes current limitations, guiding future research toward models capable of high-level system design, essential for automated software development, maintenance, and innovation in industry. It also provides a standardized platform for comparing approaches, fostering progress in AI-driven software engineering.
Technical Contribution
The paper introduces a novel evaluation framework combining automated fuzzing and behavior verification, enabling large-scale, behavior-oriented assessment of code generation models. It constructs diverse, challenging tasks from open-source repositories, emphasizing architectural decision-making. The analysis reveals model biases toward monolithic solutions and highlights the gap between current capabilities and real-world software engineering needs. The open-source dataset and testing pipeline establish a reproducible benchmark, encouraging further development of models with stronger system design skills.
Novelty
This is the first comprehensive benchmark to evaluate AI models on their ability to design and implement entire software systems based solely on executable behavior. Unlike prior benchmarks focusing on code snippets or specific tasks, ProgramBench emphasizes holistic system design, behavior verification, and architectural decision-making, representing a significant innovation in AI evaluation methodology.
Limitations
- Models tend to favor monolithic, long-function solutions, lacking modularity, which limits scalability to large, complex software projects.
- Behavioral tests, while extensive, cannot fully capture all aspects of software design and internal architecture, leading to potential overestimation of capabilities.
- Current models show limited ability to handle complex, multi-component systems, indicating the need for enhanced abstraction and planning abilities.
Future Work
Future research should explore integrating multi-modal inputs such as design diagrams and architecture specifications to improve system-level reasoning. Enhancing models' abstraction, modularization, and planning capabilities is essential. Combining reinforcement learning with behavior-driven feedback could foster better architectural decision-making. Expanding the benchmark to include more complex, real-world projects and multi-language scenarios will further push the boundaries of AI in software engineering.
AI Executive Summary
The rapid advancement of large language models (LLMs) has opened new avenues for automating software development. However, evaluating their true capability to design and build entire systems remains a challenge. Traditional benchmarks focus on code snippets or specific functions, leaving a gap in understanding how well models can handle holistic software engineering tasks. To address this, the authors introduce ProgramBench, a novel benchmark that tests models on their ability to reconstruct full software projects from minimal input—only an executable and documentation.
In this framework, models are tasked with designing architecture, modularizing code, and implementing behaviorally equivalent programs without access to source code. The evaluation employs automated fuzzing to generate behavioral tests, which verify if the reconstructed program matches the original's input-output behavior. This approach ensures that the assessment focuses on functional correctness rather than code structure, allowing for diverse solutions.
The experimental results involve nine state-of-the-art language models evaluated across 200 tasks, ranging from simple CLI tools to complex systems like FFmpeg and PHP interpreters. The findings reveal that none of the models fully resolve any task, with the best achieving only 3% of tasks with over 95% test pass rates. Models tend to produce monolithic, lengthy functions, diverging from human modular design principles. Despite this, the generated tests achieve coverage comparable to human-written tests, validating behavioral correctness.
This research underscores the significant gap between current AI capabilities and the demands of real-world software engineering. It highlights the importance of architectural decision-making and modularization, areas where models need substantial improvement. The ProgramBench benchmark provides a critical platform for future development, aiming to foster models capable of high-level system design, ultimately transforming automated software engineering and accelerating industry innovation. While promising, the work also acknowledges limitations in handling complex, multi-component systems and calls for integrating design reasoning and multi-modal inputs in future efforts.
Deep Dive
Abstract
Turning ideas into full software projects from scratch has become a popular use case for language models. Agents are being deployed to seed, maintain, and grow codebases over extended periods with minimal human oversight. Such settings require models to make high-level software architecture decisions. However, existing benchmarks measure focused, limited tasks such as fixing a single bug or developing a single, specified feature. We therefore introduce ProgramBench to measure the ability of software engineering agents to develop software holisitically. In ProgramBench, given only a program and its documentation, agents must architect and implement a codebase that matches the reference executable's behavior. End-to-end behavioral tests are generated via agent-driven fuzzing, enabling evaluation without prescribing implementation structure. Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task, with the best model passing 95\% of tests on only 3\% of tasks. Models favor monolithic, single-file implementations that diverge sharply from human-written code.