ProjectEval: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generation
ProjectEval introduces a user-interaction-based benchmark for project-level code generation evaluation, with 284 tasks and multi-level inputs, enhancing automation and explainability.
Key Findings
Methodology
This study constructs ProjectEval by integrating large language models (e.g., GPT-4o) with human review, designing three input levels—natural language prompts, checklists, and code skeletons—and simulating user interactions for evaluation. The framework employs multiple objective metrics, including Pass@K, CodeBLEU, and Levenshtein distance, to assess code correctness and similarity. The benchmark encompasses 284 real-world tasks across website and batch scenarios, enabling automated, fine-grained, and interpretable evaluation. The process involves generating solutions, converting them into executable projects, and testing through simulated user interactions, thus overcoming limitations of traditional unit tests and manual review.
Key Results
- GPT-4o achieved an average Pass@5 of 12.49% across 20 tasks, indicating significant room for improvement in complex project understanding.
- Multi-level inputs notably improved evaluation granularity and interpretability, especially in user interaction simulation scenarios.
- Findings highlight that systematic engineering comprehension and comprehensive analysis are critical for practical project success, with current models still underperforming in real-world complexity.
Significance
This work addresses the critical gap in automated, explainable, and project-level code evaluation, bridging the gap between academic benchmarks and industrial needs. By simulating user interactions, ProjectEval provides a more realistic assessment of model capabilities, facilitating development of more effective programming agents. It advances the state-of-the-art in automated software evaluation, supporting scalable, transparent, and practical deployment in real-world applications, thus promoting progress toward autonomous software engineering.
Technical Contribution
The framework introduces a multi-layer input design and user interaction simulation, coupled with multi-metric evaluation, to deliver detailed, automated assessment of code generation at the project level. It constructs a comprehensive dataset of 284 complex tasks, enabling nuanced performance analysis. The approach significantly enhances automation and explainability over existing benchmarks, setting a new standard for project-level evaluation and providing a foundation for future research in automated software testing.
Novelty
This is the first benchmark to incorporate user interaction simulation into project-level code evaluation, moving beyond static unit tests and manual review. Its multi-input, multi-metric design offers a novel, holistic assessment method that closely mirrors real-world usage, representing a significant innovation over prior benchmarks like DevBench and ProjectDev.
Limitations
- Model performance remains limited in complex, multi-task scenarios, with low success rates indicating the need for improved understanding and reasoning capabilities.
- Simulation-based evaluation may not fully capture real user behaviors, potentially introducing biases or inaccuracies.
- Test case design and task complexity are still limited; future work should include more diverse and challenging scenarios to better reflect industrial applications.
Future Work
Future directions include integrating multimodal data (e.g., visual, structural), applying reinforcement learning for better project understanding, and incorporating real user feedback to refine evaluation metrics. Expanding task diversity and complexity will further improve robustness. Additionally, optimizing the evaluation pipeline for efficiency and scalability will facilitate industrial deployment, ultimately accelerating the development of autonomous programming agents.
AI Executive Summary
The rapid advancement of large language models (LLMs) like GPT-4 and Codex has revolutionized automatic code generation. However, evaluating their performance at the project level—beyond single functions—remains a significant challenge. Traditional benchmarks such as HumanEval and MBPP mainly assess isolated functions, lacking the capacity to gauge models' understanding of complex, multi-component software projects. Moreover, existing project-level benchmarks like DevBench rely heavily on manual review or simplistic test units, which do not fully reflect real-world deployment scenarios.
Addressing this gap, our research introduces ProjectEval, a novel benchmark designed to simulate real user interactions for comprehensive project-level code evaluation. Unlike prior methods, ProjectEval employs a multi-level input system—natural language prompts, detailed checklists, and code skeletons—allowing models to generate solutions at different abstraction layers. The benchmark includes 284 real-world tasks, covering website and batch processing scenarios, with diverse and challenging test cases. Evaluation metrics combine success rates (Pass@K) with code similarity measures like CodeBLEU and Levenshtein distance, providing a multi-faceted view of model performance.
The experimental results reveal that even state-of-the-art models like GPT-4o achieve only about 12.49% Pass@5, highlighting the difficulty of project-level understanding. Multi-level inputs significantly enhance interpretability and robustness, emphasizing the importance of systematic engineering comprehension. These findings suggest that future improvements should focus on multi-modal data integration and reinforcement learning strategies to boost model capabilities.
Overall, ProjectEval offers a more realistic, automated, and explainable framework for assessing programming agents, bridging the gap between research and industrial application. It provides a scalable platform to benchmark progress, guide model development, and ultimately accelerate the deployment of autonomous coding systems in real-world environments. Despite current limitations, this work marks a critical step toward intelligent, reliable, and practical software engineering automation.
Deep Analysis
Background
The evolution of AI-driven code generation has transitioned from simple function synthesis to complex project-level development. Early benchmarks like HumanEval and MBPP focused on isolated algorithms, providing limited insight into models' holistic understanding. Recent efforts, such as DevBench and ProjectDev, introduced project-level evaluation but relied heavily on manual review or static test units, which are labor-intensive and less reflective of real-world scenarios. As models like GPT-4 and Gemma-2 improve, the need for automated, scalable, and interpretable evaluation frameworks becomes urgent. This background underscores the importance of developing benchmarks that simulate actual user workflows, enabling more meaningful assessment and guiding future model improvements.
Core Problem
Current evaluation methods struggle to accurately measure models' ability to handle complex, multi-component projects with real user interactions. Manual reviews are subjective and laborious, while static test units do not capture dynamic user behaviors. Consequently, there is a gap in assessing models' understanding, analysis, and engineering skills in realistic settings. The challenge lies in designing an automated, explainable, and scalable benchmark that can evaluate models across diverse project scenarios, providing actionable insights for development and deployment in industrial contexts.
Innovation
The key innovations include: 1) multi-level input design—natural language prompts, checklists, and code skeletons—allowing layered understanding and solution generation; 2) user interaction simulation, enabling evaluation of models' ability to handle real-world workflows; 3) multi-metric evaluation combining success rate with code similarity measures, enhancing interpretability; 4) extensive dataset of 284 real-world tasks, covering diverse scenarios, providing a comprehensive testing ground. These innovations collectively enable a more realistic, automated, and detailed assessment of project-level code generation, surpassing existing benchmarks in granularity and applicability.
Methodology
- �� Task collection: 20 complex real-world projects sourced from public repositories and custom design, with detailed descriptions and test cases.
- �� Input design: three levels—Level 1 (natural language prompt), Level 2 (detailed checklist), Level 3 (code skeleton)—generated via large models and human review.
- �� Solution generation: models produce code solutions based on input levels, which are then converted into executable projects.
- �� User interaction simulation: automated scripts mimic user behaviors such as browsing, clicking, and command input to test the generated projects.
- �� Evaluation: success measured by Pass@K, supplemented with code similarity metrics (CodeBLEU, Levenshtein), and analysis of multi-metric scores.
- �� Human review: manual validation of canonical solutions and test cases ensures robustness and interpretability.
Experiments
The evaluation involved 284 tasks across website and batch scenarios, with models like GPT-4o, Llama, and Gemma. Metrics included Pass@K (average 14.2 test cases passed per task), code similarity scores, and detailed analysis of input level effects. Experiments tested different input configurations, ablation studies on multi-level inputs, and cross-model comparisons. Results demonstrated that multi-level inputs improve interpretability and robustness, while current models still face significant challenges in understanding complex project requirements. The setup validated the benchmark's ability to distinguish model capabilities and guide future improvements.
Results
GPT-4o achieved an average Pass@5 of 12.49%, indicating substantial room for enhancement. Multi-level input strategies significantly improved evaluation granularity, especially in simulating user interactions. The results reveal that systematic engineering understanding and comprehensive analysis are essential for practical project success, with current models showing limited performance in complex, multi-component scenarios. These findings highlight the importance of integrating multi-modal data and advanced reasoning techniques for future progress.
Applications
The benchmark can be employed by industry practitioners to evaluate and improve AI-powered programming tools, facilitating automated project development and testing. It supports continuous integration workflows, enabling real-time performance monitoring and model fine-tuning. Additionally, it can serve as a training and benchmarking platform for developing more robust, explainable, and user-aligned AI coding agents, accelerating their deployment in enterprise environments.
Limitations & Outlook
Despite its advances, the benchmark still faces challenges: models perform poorly on highly complex, multi-task projects; simulated user interactions may not fully replicate real user behaviors; and the cost and time of evaluation remain high. Future work should focus on expanding task diversity, optimizing evaluation efficiency, and integrating real user feedback to enhance reliability and scalability.
Plain Language Accessible to non-experts
想象你在厨房里做饭。以前，厨师自己尝试菜味，或者用简单的味道测试来判断菜是否好吃。现在，假设你有一个虚拟厨师助手，它不仅能按照食谱做菜，还能模拟你请朋友试吃，告诉你菜的味道和外观是否符合期待。这个助手还能根据你的反馈调整菜谱，帮你做出更好吃的菜。通过这种方式，你不用亲自试吃，也能知道菜的质量。这就像是让AI帮你写代码，它不仅要完成任务，还要像真实用户一样测试和评价，确保软件能真正用在生活中。这种方法让评估变得更真实、更智能，也更容易理解和改进。
ELI14 Explained like you're 14
想象你在学校的科学实验室里做实验。以前，老师只会看你做得对不对，或者用一些简单的测试来检查。而现在，你有一个超级智能机器人助手，它不仅会帮你按照说明做实验，还会模拟你在实验中遇到的问题，比如漏掉步骤或做错了。这个机器人还能告诉你哪里做得好，哪里还可以改进。这样，你就可以更快、更准确地知道实验结果，也能学到更多知识。这个研究就像是让机器人帮你评估你的“代码”——不仅要能跑起来，还要符合用户的需求。这样，软件开发就像科学实验一样，有了更科学、更自动化的评估方法。
Abstract
Recently, LLM agents have made rapid progress in improving their programming capabilities. However, existing benchmarks lack the ability to automatically evaluate from users' perspective, and also lack the explainability of the results of LLM agents' code generation capabilities. Thus, we introduce ProjectEval, a new benchmark for LLM agents project-level code generation's automated evaluation by simulating user interaction. ProjectEval is constructed by LLM with human reviewing. It has three different level inputs of natural languages or code skeletons. ProjectEval can evaluate the generated projects by user interaction simulation for execution, and by code similarity through existing objective indicators. Through ProjectEval, we find that systematic engineering project code, overall understanding of the project and comprehensive analysis capability are the keys for LLM agents to achieve practical projects. Our findings and benchmark provide valuable insights for developing more effective programming agents that can be deployed in future real-world production.