Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios
Introduced CLI-Tool-Bench, a black-box differential testing framework, achieving only 43.8% success in end-to-end CLI tool generation, highlighting ongoing challenges.
Key Findings
Methodology
This study constructs CLI-Tool-Bench, utilizing 94 real-world CLI repositories across Python, JavaScript, and Go. It employs automated schema extraction, LLM-guided fuzzing for test case generation, and black-box differential testing to evaluate behavior consistency. The evaluation metrics include execution success, side-effect consistency, and behavioral equivalence, avoiding reliance on predefined structures. Experiments involve seven state-of-the-art LLMs tested under two agent frameworks, with success rates capped at 43.8%, revealing significant challenges in autonomous code generation.
Key Results
- Kimi-k2.5 outperforms others with a maximum success rate of 43.78%, significantly below human performance. Models tend to produce monolithic code structures, and higher token consumption does not necessarily improve outcomes. Framework differences influence results notably, emphasizing the importance of behavior-based evaluation.
- Across multiple languages and complexity levels, success drops sharply for harder tasks. The experiments also expose deficiencies in dependency management and error handling, underscoring the importance of black-box, behavior-focused testing.
- Comparison with white-box testing shows that behavior-oriented black-box evaluation better reflects real user scenarios, guiding future assessment methodologies.
Significance
This work advances beyond traditional structure-dependent and white-box assessments, proposing a behavior-centric, structure-agnostic evaluation framework that aligns more closely with real-world user expectations. The findings demonstrate that, despite progress in code completion, full end-to-end software generation remains a formidable challenge. This has profound implications for AI-assisted development, pushing the community toward more realistic benchmarks and robust models that can independently plan and execute software projects.
Technical Contribution
The paper introduces a novel, structure-agnostic evaluation pipeline combining schema extraction, LLM-guided fuzzing, and black-box differential testing. This approach enables behavior-level validation without relying on predefined repository layouts, representing a significant departure from prior white-box-centric methods. The framework supports comprehensive, automated assessment of generated software in realistic scenarios, providing a new standard for evaluating autonomous code generation systems.
Novelty
This is the first framework to evaluate full CLI tool generation in a structure-agnostic, end-to-end manner using black-box differential testing. Unlike previous benchmarks that depend on fixed templates or white-box tests, this approach emphasizes behavior verification, making it more applicable to real-world scenarios where internal code details are unknown. It sets a new benchmark for autonomous software engineering evaluation.
Limitations
- The current framework focuses solely on CLI tools, limiting its applicability to GUI or web applications. Extending to more complex, interactive systems remains future work.
- Models still struggle with dependency resolution and multi-module architectures, indicating room for improvement in structural planning capabilities.
- The dataset relies on high-quality open-source repositories, which may not fully represent proprietary or less-maintained projects. Broader data collection is needed.
Future Work
Future directions include integrating reinforcement learning and explicit structural planning to enhance autonomy. Incorporating multimodal inputs like design diagrams or natural language specifications could further improve understanding. Additionally, refining evaluation metrics to include user experience and performance will promote more practical AI-assisted software development.
AI Executive Summary
The rapid advancement of large language models (LLMs) has revolutionized automated code generation, yet evaluating their true end-to-end capabilities remains a challenge. Traditional benchmarks like HumanEval or MBPP focus on function-level tasks, and even repository-level benchmarks such as SWE-bench rely heavily on predefined structures and white-box testing, which do not accurately reflect real-world scenarios. To address this gap, this study introduces CLI-Tool-Bench, a novel evaluation framework that emphasizes behavior-based, structure-agnostic assessment of CLI tool generation.
By curating a diverse dataset of 94 real-world CLI repositories across Python, JavaScript, and Go, and employing automated schema extraction combined with LLM-guided fuzzing, the framework generates comprehensive test cases. These are executed in isolated Docker environments, with results compared via black-box differential testing that measures execution success, system side effects, and output semantics. This approach ensures that the evaluation closely mimics user interactions, focusing on the external behavior of the generated tools.
Experimental results across seven state-of-the-art models reveal a maximum success rate of only 43.8%, underscoring the significant difficulty of true zero-to-one software generation. The models tend to produce monolithic code structures, and increased token consumption does not necessarily translate into better performance. The findings highlight the importance of behavior-centric evaluation and suggest that current models still require substantial improvements in structural planning and robustness.
This work pushes the frontier of autonomous software engineering, advocating for more realistic, behavior-focused benchmarks. It also opens avenues for future research integrating structural planning, multimodal inputs, and user-centric metrics, ultimately aiming to realize AI systems capable of independently designing and deploying complete, reliable software solutions.
Deep Dive
Abstract
The evolution of Large Language Models (LLMs) has catalyzed a paradigm shift towards intent-driven software development, where autonomous agents are expected to design and deliver complete, runnable software systems from scratch. However, existing benchmarks fail to adequately assess this 0-to-1 generation capability due to two fundamental limitations. First, they rely on predefined structural scaffolds, which reduces the task to mere file-filling. Second, they depend on rigid white-box unit testing, which forces generated code to conform to specific internal implementations rather than validating end-to-end user-centric behavior. To bridge this gap, we introduce CLI-Tool-Bench, a novel, structure-agnostic benchmark designed to evaluate the ground-up generation of Command-Line Interface (CLI) tools. Powered by an automated black-box differential testing framework, the benchmark comprises 94 high-quality, real-world repositories spanning diverse programming languages and complexity levels. For each task, agents are provided with an empty workspace, forcing them to autonomously handle repository planning and dependencies. We evaluate the generated software by executing it in isolated sandboxes. The system-level side effects and terminal outputs are then compared against human-written oracles using a rigorous multi-tiered equivalence metric. Extensive evaluation of seven state-of-the-art LLMs reveals that the top-tier models achieve a maximum overall success rate of only 43.8%, highlighting that 0-to-1 software generation remains a highly challenging frontier. Furthermore, we discover that agents exhibit a strong tendency to generate monolithic code structures, and that higher token consumption does not necessarily yield better task performance.