FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation
FEA-Bench evaluates LLMs' ability for repository-level feature development; current models resolve only ~10% of tasks.
Key Findings
Methodology
This study constructs a dataset from 83 GitHub repositories' pull requests focused on new feature addition. Using rule-based filtering and GPT-4-based intent classification, tasks are selected where code changes include new components paired with unit tests. Models like GPT-4, CodeLlama, and DeepSeek-Coder are evaluated under various prompts and retrieval strategies, with success measured by task resolution and test pass rates. The pipeline ensures diversity and verification through automated environment setup and code application, emphasizing real-world complexity.
Key Results
- The best model, DeepSeek-R1, achieves only about 9.92% task resolution rate under Oracle conditions, indicating significant difficulty in repository-level incremental feature development.
- Larger models tend to perform better, with Qwen2.5-Coder and GPT-4 approaching 4-6% resolution in some settings, yet still far from practical deployment levels.
- Enhanced prompts and retrieval methods slightly improve performance, but overall, models struggle with understanding multi-file, multi-step tasks, revealing substantial gaps in current capabilities.
Significance
This work pioneers a systematic benchmark for evaluating large models' competence in implementing new features at the repository level, a core aspect of software engineering. The results highlight the gap between current model performance and real-world requirements, guiding future research toward models capable of understanding complex, multi-file projects and performing multi-step edits. It addresses industry needs for automated feature development, potentially transforming software workflows and reducing manual effort in large-scale projects.
Technical Contribution
The paper introduces a novel dataset and evaluation framework for repository-level incremental development, combining automated data collection, intent classification, and execution-based validation. It extends beyond traditional code completion benchmarks by focusing on multi-file, multi-step feature implementation, and provides a comprehensive analysis of model performance across diverse settings. This approach offers a new standard for assessing and advancing AI capabilities in real-world software engineering tasks.
Novelty
This is the first benchmark specifically targeting repository-level feature implementation, leveraging pull request data to simulate real development scenarios. Unlike prior benchmarks focused on isolated code snippets or bug fixes, FEA-Bench emphasizes multi-file, multi-step tasks with verification via unit tests. Its combination of automatic data curation, intent classification, and execution validation marks a significant innovation in AI-driven software engineering evaluation.
Limitations
- Models show limited understanding of long-range context and multi-file dependencies, constrained by training data and architecture limitations.
- Evaluation is based on single-round generation, lacking multi-turn interaction modeling essential for real-world development.
- Dataset predominantly covers Python projects, limiting cross-language applicability and generalization.
Future Work
Future efforts will explore multi-turn, continuous development scenarios, integrating reinforcement learning and multimodal data to enhance understanding of complex repositories. Expanding to other programming languages and incorporating multi-step, multi-file workflows will further bridge the gap between current models and practical software engineering needs.
AI Executive Summary
The rapid advancement of large language models (LLMs) like GPT-4 and CodeLlama has revolutionized code generation, offering promising tools for automating software development. However, most existing benchmarks focus on isolated tasks such as function synthesis or bug fixing, which do not reflect the complexity of real-world projects involving multi-file, multi-step feature development. Recognizing this gap, this study introduces FEA-Bench, a novel benchmark designed to evaluate LLMs' capabilities in repository-level incremental feature implementation.
FEA-Bench is constructed from pull request data across 83 diverse GitHub repositories, focusing on tasks where new components are added to implement features. The dataset employs rule-based filtering to select relevant pull requests, combined with GPT-4-based intent classification to ensure the tasks are genuinely feature-oriented. Each task includes code changes and associated unit tests, enabling execution-based validation of model outputs. This setup simulates real development workflows, requiring models to generate new code and modify existing code across multiple files.
The evaluation involves multiple models, including GPT-4, CodeLlama, and DeepSeek-Coder, tested under various prompt and retrieval strategies. Results reveal that the best-performing models resolve only around 10% of tasks, highlighting significant challenges in understanding complex codebases, long contexts, and multi-file dependencies. Larger models tend to perform slightly better, but the overall performance remains far from practical deployment standards.
These findings underscore the need for further research into models capable of multi-step, multi-file reasoning and editing. The benchmark provides a critical foundation for future work, guiding efforts to develop AI systems that can truly support automated, large-scale software development. Despite current limitations, this work marks an important step toward intelligent, autonomous software engineering, with broad implications for industry and academia. Continued improvements in model architecture, training data diversity, and multi-turn interaction will be essential to realize this vision.
Deep Analysis
Background
The evolution of automated software engineering has transitioned from static analysis tools to deep learning-based code generation models. Early systems like static analyzers addressed bug detection and code quality, but lacked generative capabilities. Recent breakthroughs with models such as GPT-4, Codex, and CodeLlama have demonstrated impressive performance in code synthesis, debugging, and translation, leading to the rise of developer assistants like Copilot. Despite these advances, existing benchmarks such as HumanEval, MBPP, and CodeX primarily evaluate isolated code snippets, failing to capture the complexity of real-world projects involving multiple files, steps, and collaborative workflows. Recent efforts like SWE-bench and DevEval have begun to address repository-level tasks, mainly focusing on bug fixing and code repair, but lack comprehensive evaluation of feature addition tasks. As software projects grow in size and complexity, there is an urgent need for benchmarks that reflect the true challenges faced by developers, including incremental feature development, multi-file coordination, and multi-step reasoning.
Core Problem
Current large language models exhibit limited capacity to perform repository-level incremental feature development, especially in complex, multi-file environments. Existing benchmarks do not adequately measure the ability to generate new components, modify existing code, and verify changes within a large codebase. This gap hampers the development of models capable of supporting real-world software engineering tasks, which require understanding long-range dependencies, multi-file interactions, and multi-step workflows. The challenge is compounded by the difficulty in constructing representative datasets that accurately reflect the diversity and complexity of actual projects, as well as the need for reliable validation through unit tests. Addressing these issues is critical for advancing AI-assisted software engineering towards practical deployment.
Innovation
This work introduces a novel benchmark, FEA-Bench, specifically targeting repository-level incremental feature development. It leverages pull request data from diverse GitHub repositories, applying rule-based filtering to identify feature addition tasks. An intent classifier based on GPT-4 further refines task selection, ensuring focus on genuine feature implementation. The dataset includes code changes paired with unit tests, enabling execution-based validation. The evaluation framework tests multiple large models under various prompts and retrieval strategies, providing a comprehensive assessment of their ability to generate and modify code in complex, multi-file scenarios. This approach extends beyond traditional code completion, emphasizing multi-step reasoning, multi-file editing, and real-world applicability, thus representing a significant methodological advancement.
Methodology
- �� Data Collection: Automatically crawl pull requests from 83 GitHub repositories, focusing on those involving code changes to add new features.
- �� Filtering: Use rule-based criteria to select pull requests with significant new components, ensuring they account for over 25% of code edits.
- �� Intent Classification: Employ GPT-4 to classify pull request descriptions, retaining only those labeled as 'new feature'.
- �� Task Construction: Parse code diffs to extract new components, pairing them with relevant unit tests.
- �� Validation: Run unit tests before and after applying code changes to verify correctness.
- �� Model Evaluation: Test models like GPT-4, CodeLlama, DeepSeek-Coder under different prompts (detailed/brief) and retrieval methods (Oracle/BM25), measuring task resolution and test pass rates.
- �� Analysis: Compare performance across settings, analyze failure modes, and identify improvement pathways.
Experiments
The dataset comprises 1401 tasks from 83 repositories, covering diverse feature development scenarios. Models evaluated include GPT-4, CodeLlama, Qwen2.5-Coder, and DeepSeek-R1, tested with various prompt strategies and retrieval methods. Metrics include task resolution rate, unit test pass rate, and retrieval recall. Experiments explore different context lengths, output formats, and retrieval strategies, providing insights into model capabilities and limitations. Multiple ablation studies assess the impact of prompt detail, retrieval quality, and context size, revealing that current models struggle with long-range dependencies and multi-file edits. Results highlight the need for improved understanding of complex codebases and multi-step reasoning in AI models.
Results
The highest task resolution rate achieved was approximately 10% with DeepSeek-R1 under Oracle conditions, indicating substantial room for improvement. Larger models like GPT-4 and DeepSeek-V3 perform better than smaller ones but still fail to meet practical standards. Prompt design and retrieval strategies influence performance, with detailed prompts and Oracle retrieval yielding marginal gains. The results demonstrate that current models lack robust understanding of multi-file, multi-step tasks, emphasizing the necessity for architectural and training enhancements to bridge this gap. These findings provide a clear benchmark for future research aiming to elevate AI capabilities in real-world software engineering.
Applications
This benchmark can guide the development of AI tools for automating feature addition in large codebases, reducing manual effort, and accelerating software deployment cycles. It supports research into models capable of multi-file reasoning, multi-step editing, and long-context understanding, essential for industry-scale projects. Practical applications include automated code review, refactoring, and continuous integration workflows, ultimately leading to more autonomous and efficient software development pipelines.
Limitations & Outlook
Models currently exhibit limited understanding of long-range dependencies and multi-file interactions, constrained by training data and model architecture. The evaluation focuses on single-round generation, lacking multi-turn development scenarios. Dataset bias towards Python limits cross-language generalization. Additionally, the computational cost of large models restricts practical deployment. Future work should address multi-turn reasoning, multi-language support, and more efficient architectures to overcome these limitations.
Plain Language Accessible to non-experts
想象你在厨房里准备一道复杂的菜肴。你需要准备各种食材，按照特定的步骤把它们组合在一起，还要确保每一步都正确。现在，假设你有一个聪明的助手，它可以帮你准备食材、告诉你下一步怎么做，还能帮你调整菜谱。刚开始，它可能只会帮你切菜（补全代码），但随着学习，它能帮你设计新菜（新功能），甚至帮你改良菜谱（编辑代码）。这个助手就像大模型一样，能理解你要做的菜（功能），帮你一步步完成。软件开发也是这样，程序员不断添加新功能，修改旧代码，助手也在学习如何更好地帮忙，让软件变得更厉害。
ELI14 Explained like you're 14
想象你在学校的科学实验室里做实验。每次你都需要准备不同的材料，按照步骤操作，最后得到一个实验结果。有时候，你的老师会让你试试不同的方法，或者改进实验步骤。现在，假设你有一个聪明的机器人助手，它可以帮你准备材料、告诉你下一步怎么做，还能帮你改进实验方案。刚开始，它可能只会帮你把材料准备好（生成代码），但随着时间，它能帮你设计新实验（新功能），甚至帮你改正错误（编辑代码）。这个机器人就像大模型一样，能理解你要做的事情，帮你一步步完成。软件开发也是这样，程序员不断添加新功能，修改旧代码，机器人助手也在学习如何更好地帮忙，让软件变得更厉害。
Abstract
Implementing new features in repository-level codebases is a crucial application of code generation models. However, current benchmarks lack a dedicated evaluation framework for this capability. To fill this gap, we introduce FEA-Bench, a benchmark designed to assess the ability of large language models (LLMs) to perform incremental development within code repositories. We collect pull requests from 83 GitHub repositories and use rule-based and intent-based filtering to construct task instances focused on new feature development. Each task instance containing code changes is paired with relevant unit test files to ensure that the solution can be verified. The feature implementation requires LLMs to simultaneously possess code completion capabilities for new components and code editing abilities for other relevant parts in the code repository, providing a more comprehensive evaluation method of LLMs' automated software engineering capabilities. Experimental results show that LLMs perform significantly worse in the FEA-Bench, highlighting considerable challenges in such repository-level incremental code development.