HumanEvo: An Evolution-aware Benchmark for More Realistic Evaluation of Repository-level Code Generation
HumanEvo benchmark models software evolution, revealing performance overestimation by 10-61% in traditional evaluations.
Key Findings
Methodology
The study constructs HumanEvo, a dataset simulating software project evolution by rolling back repositories to pre-commit states before target functions. It employs automated execution-based evaluation, categorizes functions by dependency levels, and tests seven diverse large language models (LLMs) including GPT-4, CodeLlama, and DeepSeekCoder. The evaluation compares traditional latest-version context methods with evolution-aware settings, analyzing performance drops across dependency complexities and project stages. Results show that ignoring evolution inflates performance metrics by 10-61%, especially for functions with complex dependencies, highlighting the importance of temporally accurate context in realistic assessments.
Key Results
- All evaluated models show significant performance decline (~30%) when tested in evolution-aware settings, confirming previous overestimations.
- Functions with inter-class dependencies exhibit a 30.9% success rate decrease compared to simpler functions, indicating dependency complexity impacts model accuracy.
- Performance degradation correlates with project evolution, with models gradually diverging from true capabilities over time, emphasizing the need for temporally aligned evaluation.
Significance
This work underscores the critical need to incorporate software evolution into code generation benchmarks, addressing the prevalent overestimation bias. It provides a more realistic framework for evaluating models' practical utility in maintenance and development tasks, fostering trust and guiding improvements. By revealing how dependency complexity and project timeline influence performance, it informs future model design and deployment strategies, ultimately advancing AI-assisted software engineering and industry adoption.
Technical Contribution
The paper introduces HumanEvo, a novel evolution-aware benchmark combining repository rollback, dependency classification, and automated testing. It pioneers the systematic quantification of evolution-induced performance bias, establishing a new standard for realistic evaluation. The shared toolbox and detailed methodology facilitate community adoption, enabling more accurate performance measurement of LLMs in dynamic software environments. This approach bridges the gap between static benchmarks and real-world development scenarios, offering theoretical insights and practical tools for AI in software engineering.
Novelty
This is the first benchmark explicitly modeling software project evolution for repository-level code generation, moving beyond static snapshots. Its integration of version rollback, dependency stratification, and automated validation creates a realistic simulation of development processes, setting a new standard in evaluation methodology. This innovation addresses a long-standing gap, providing a more truthful measure of models' capabilities in practical, evolving environments.
Limitations
- Currently limited to Python and Java, future work should include more languages and multi-language projects.
- Dependency classification, while detailed, does not fully capture complex multi-dependency scenarios in large-scale projects.
- Extreme cases of project evolution, such as rapid or drastic changes, require further investigation to understand their impact on model performance.
Future Work
Future research will extend the benchmark to additional languages, incorporate more complex dependency structures, and simulate more aggressive evolution scenarios. Developing adaptive models that can handle rapid project changes and integrating real-time version control data will further enhance evaluation realism. Additionally, establishing industry standards for evolution-aware benchmarking can promote broader adoption, ultimately improving the robustness and reliability of AI tools in software engineering.
AI Executive Summary
The rapid advancement of large language models (LLMs) has revolutionized automated code generation, promising to accelerate software development and maintenance. However, most existing evaluation methods rely on static snapshots of code repositories, typically using the latest version as context. This approach neglects the inherently dynamic nature of software projects, which evolve over time through feature additions, bug fixes, and dependency updates. As a result, performance assessments based solely on the latest code tend to overstate the models' real-world capabilities, leading to inflated metrics that do not reflect practical scenarios.
Recognizing this gap, the authors introduce HumanEvo, a pioneering benchmark designed to incorporate software evolution into repository-level code generation evaluation. HumanEvo simulates the development process by rolling back repositories to states prior to the target function's creation, ensuring that the context provided to models mirrors what developers would have had at that time. This process involves meticulous data collection from high-quality open-source projects in Python and Java, filtering pull requests through rigorous testing and dependency analysis, and manually rewriting docstrings to diversify input styles.
The evaluation compares seven diverse models, including GPT-4, CodeLlama, and DeepSeekCoder, across 400 tasks categorized by dependency levels. Results reveal a stark performance decline—up to 61.1%—when models are tested in an evolution-aware setting, highlighting the overestimation bias in traditional benchmarks. Functions with complex dependencies suffer more, with success rates dropping significantly, especially in inter-class dependency scenarios. These findings demonstrate that ignoring software evolution leads to overly optimistic assessments, which could misguide model development and deployment.
This work has profound implications for both academia and industry. It emphasizes the necessity of realistic evaluation frameworks that account for project timelines and dependency complexities, ultimately fostering more robust and applicable AI tools. The authors also provide an open-source toolbox to facilitate community adoption and future research. Moving forward, expanding the benchmark to include more languages, complex dependencies, and real-time evolution scenarios will further enhance its utility. Overall, HumanEvo sets a new standard for evaluating AI in software engineering, aligning performance metrics with real-world development challenges and paving the way for more trustworthy AI-assisted coding solutions.
Deep Dive
Plain Language Accessible to non-experts
想象你在一家工厂工作，这个工厂每天都在变化。有人加入新机器，旧机器被拆掉，生产流程也在调整。你需要不断适应这些变化，才能确保工厂正常运转。软件项目就像这个工厂，代码、依赖关系、功能都在不断变化。传统的评估方法就像只看工厂的最后样子，忽略了这些变化，可能会觉得工厂很厉害，但实际上你不知道它在变化过程中遇到的困难。HumanEvo就像是让你回到过去某个时间点，看看那时的工厂是怎样的，然后测试你在那个时间点的工作能力。这样评估才更真实，也更公平。它帮助我们理解，软件像工厂一样，只有考虑了变化，才能真正知道模型的能力。
ELI14 Explained like you're 14
想象你在学校参加一个比赛，你的老师让你用不同的材料做一件作品。但是，老师只让你看最后完成的作品，没有告诉你在制作过程中用了哪些材料和步骤。你可能会觉得这个作品很棒，但其实你不知道在制作过程中遇到了什么困难，也不知道用了哪些特别的材料。现在，如果老师让你回到制作的中间阶段，看看当时用的材料和步骤，你就能更清楚这个作品是怎么做出来的。这个就像HumanEvo，它让我们回到软件开发的不同时间点，看看当时的代码和依赖关系，这样评估模型的能力才会更真实、更公平。否则，只看最后的结果，就像只看最后的作品，可能会误导我们对过程的理解。
Abstract
To evaluate the repository-level code generation capabilities of Large Language Models (LLMs) in complex real-world software development scenarios, many evaluation methods have been developed. These methods typically leverage contextual code from the latest version of a project to assist LLMs in accurately generating the desired function. However, such evaluation methods fail to consider the dynamic evolution of software projects over time, which we refer to as evolution-ignored settings. This in turn results in inaccurate evaluation of LLMs' performance. In this paper, we conduct an empirical study to deeply understand LLMs' code generation performance within settings that reflect the evolution nature of software development. To achieve this, we first construct an evolution-aware repository-level code generation dataset, namely HumanEvo, equipped with an automated execution-based evaluation tool. Second, we manually categorize HumanEvo according to dependency levels to more comprehensively analyze the model's performance in generating functions with different dependency levels. Third, we conduct extensive experiments on HumanEvo with seven representative and diverse LLMs to verify the effectiveness of the proposed benchmark. We obtain several important findings through our experimental study. For example, we find that previous evolution-ignored evaluation methods result in inflated performance of LLMs, with performance overestimations ranging from 10.0% to 61.1% under different context acquisition methods, compared to the evolution-aware evaluation approach. Based on the findings, we give actionable suggestions for more realistic evaluation of LLMs on code generation. We also build a shared evolution-aware code generation toolbox to facilitate future research.