Breaking, Stale, or Missing? Benchmarking Coding Agents on Project-Level Test Evolution

The evolution of software systems necessitates continuous updates to test suites to maintain reliability and coverage. Prior research has largely focused on method-level test evolution, employing benchmarks like Defects4J, which simplify the problem by pre-pairing tests with specific methods. However, real-world projects involve complex, project-wide changes affecting multiple modules and tests across different files. Recent advances in large language models and autonomous coding agents have shown promise, but their evaluation remains limited to simplified scenarios. Existing benchmarks such as SITAR, CHOSEN, and Reaccept do not capture the full complexity of project-level test evolution, especially the identification of affected tests and the generation of new tests for new behaviors. This gap underscores the need for a comprehensive, project-level benchmark that can evaluate AI systems' ability to handle real-world code changes and test maintenance challenges.

The core challenge in test evolution is twofold: first, accurately identifying which tests are affected by code changes across a large codebase; second, generating or updating tests to reflect new or modified behaviors. Existing benchmarks bypass the identification step by pre-pairing tests with code, limiting their applicability to real-world scenarios where developers must locate affected tests manually. Moreover, current models predominantly focus on fixing failing tests (Test-Breaking), neglecting the more subtle issues of stale tests that no longer validate intended behaviors, and completely missing new behaviors that lack tests. This incomplete coverage hampers the development of truly autonomous test maintenance systems capable of handling the full spectrum of test evolution in complex projects.

This work introduces a comprehensive project-level test evolution task, emphasizing autonomous identification of affected tests, localization of new test needs, and generation of test patches. It extends prior method-level approaches by considering the entire project context, capturing three evolution types—Test-Breaking, Test-Stale, and Test-Missing—thus providing a more realistic and complete evaluation framework. The construction of TEBench from real-world open-source projects, with detailed annotations, enables systematic assessment of AI models' capabilities in complex scenarios. Additionally, the evaluation of multiple large language models across different frameworks reveals inherent performance ceilings and highlights the need for deeper semantic reasoning, guiding future research directions.

• �� Data Collection: Extract 67,670 commits from 14 Maven-based Java projects in Defects4J, applying static and execution-based filters to select high-quality instances.
• �� Filtering: Remove commits without substantive code changes, unrelated modifications, or lacking impact on tests, resulting in 314 instances across 10 projects.
• �� Version Structuring: Create three versions per instance—V−1 (pre-change), V−0.5 (post-change without test updates), V0 (full change with test modifications)—for evaluation.
• �� System Evaluation: Deploy seven configurations spanning three frameworks and six models, assessing identification via precision, recall, F1, and update quality via executability, coverage, and similarity metrics.
• �� Metrics: Use project-level autonomous identification, compare detected affected tests with ground truth, and evaluate generated test patches for correctness and semantic alignment.

The experimental setup involves evaluating models on the curated TEBench dataset, measuring identification accuracy (F1 scores) and update quality (executability, coverage, similarity). Baselines include heuristic methods and various large language models like Codex, Claude, and OpenCode. Metrics include precision, recall, F1, and structural dependency analysis. Hyperparameters such as beam size, temperature, and prompt design are tuned for optimal performance. Ablation studies analyze the impact of different input representations and reasoning modules. The evaluation also examines the difficulty of each test evolution type, revealing the models' reliance on failure signals and their limited semantic reasoning capacity.

Results show all configurations achieve F1 scores between 45.7% and 49.4%, with minimal variation across frameworks and models, indicating a performance ceiling. Test-Stale detection remains challenging, with an average F1 of about 36%, due to reliance on execution failure signals and lack of semantic understanding. Generated patches are highly executable but often differ in surface form from Ground Truth, highlighting limitations in semantic alignment. Structural dependency analysis recovers approximately 66% of affected tests, leaving a significant portion undetected. These findings point to the necessity of integrating deeper semantic reasoning and multi-modal data to improve test identification and generation.

The benchmark facilitates the development of AI systems capable of autonomously maintaining large-scale test suites in complex software projects. Immediate applications include automated test repair, regression detection, and test augmentation in CI/CD pipelines. Long-term, the research aims to create intelligent testing agents that can understand code semantics, adapt to evolving software requirements, and reduce manual effort in test maintenance, significantly improving software reliability and developer productivity across industries.

Current models heavily depend on execution failure signals, limiting detection of semantically outdated or missing tests. The lack of proactive semantic reasoning hampers performance on Stale and Missing categories. Generated patches often diverge from developer intent, indicating limited understanding of test semantics. The evaluation scope is limited to Java projects and specific frameworks, affecting generalizability. Computational costs and the complexity of integrating deeper semantic modules also pose challenges for practical deployment.