SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
SlopCodeBench evaluates how coding agents' code quality degrades over long iterative tasks, revealing increasing code bloating and complexity, with a maximum solve rate of 14.8%.
Key Findings
Methodology
This study introduces SlopCodeBench, a language-agnostic benchmark comprising 36 software development tasks and 196 checkpoints designed to evaluate how code evolves as agents repeatedly extend their own solutions. Each task specifies only external behavior via CLI or API, leaving internal architecture unconstrained. The benchmark measures two key quality metrics across trajectories: structural erosion, indicating the concentration of complexity in functions, and verbosity, reflecting redundant or bloated code. The evaluation involves 15 models, including open-source and proprietary systems, analyzing their performance over multiple iterations. The study compares agent-generated code with 473 open-source Python repositories, revealing that agents produce code that is 2.3× more verbose and 2.0× more structurally eroded. Prompt engineering strategies, such as anti-slop and plan-first prompts, can reduce initial code quality issues but do not prevent ongoing degradation, which occurs at an average rate of 1.3% per checkpoint.
Key Results
- No model fully solves any problem end-to-end; the best model achieves only 14.8% solution rate across 196 checkpoints. Structural erosion increases in 77% of trajectories, and verbosity rises in 75.5%. Compared to human repositories, agent code is significantly more bloated and complex, with degradation rates roughly double that of human code. Prompt strategies can lower initial issues by up to 62.3% for erosion and 34.8% for verbosity but fail to halt the long-term decline. The cost per checkpoint increases by 12.1% with prompt modifications, and correctness slightly drops by 2.3 percentage points.
- Model performance analysis shows that GPT-5.5 achieves the highest strict solve rate at 14.8%, yet overall performance remains far from perfect. The degradation trend is consistent across different models, with structural erosion and verbosity increasing in most trajectories. The comparison with open-source repositories underscores the risk of code quality deterioration in automated systems, especially over multiple iterations.
- Prompt engineering techniques such as anti-slop prompts effectively reduce initial code issues but do not prevent ongoing degradation. These strategies, while beneficial initially, lead to higher costs and marginally lower correctness over time. The experiments highlight the importance of architectural improvements and quality-guided training to mitigate long-term code deterioration.
- The study emphasizes that current models, despite promising capabilities, are prone to accumulating code complexity and redundancy during iterative development. This poses challenges for deploying AI in real-world software engineering tasks that require sustained, maintainable codebases. The benchmark provides a foundation for future research aimed at enhancing model robustness and long-term code quality.
- Overall, SlopCodeBench offers a novel, systematic approach to measuring long-term code degradation, revealing critical insights into the limitations of current AI coding agents. It underscores the necessity of integrating architectural and training strategies to ensure sustainable, high-quality automated software development.
Significance
This research marks a significant step forward in understanding the limitations of AI-driven code generation over extended development cycles. By quantifying how code quality deteriorates through structural erosion and verbosity, it exposes fundamental challenges faced by current models in maintaining sustainable, extensible codebases. The benchmark's design, focusing solely on external behavior and hiding internal architecture, closely mirrors real-world software engineering, making its findings highly relevant for industry applications. The insights gained highlight the urgent need for improved model architectures, quality-guided training, and better prompt strategies to prevent long-term code degradation. This work thus bridges a critical gap between short-term correctness assessments and the long-term sustainability of AI-generated code, providing a valuable tool for researchers and practitioners aiming to develop more reliable, maintainable AI coding systems.
Technical Contribution
The primary technical contribution of this work is the development of SlopCodeBench, a comprehensive, language-agnostic benchmark that evaluates long-term code evolution in AI agents. It introduces two novel metrics—structural erosion, based on cyclomatic complexity concentration, and verbosity, based on code duplication and redundant patterns—to quantify code quality degradation over iterative extensions. The benchmark's design emphasizes external behavior specifications, with internal architecture left unconstrained, enabling realistic assessment of architectural decision-making. The evaluation of 15 models across diverse tasks demonstrates the persistent challenge of code deterioration, providing empirical data on the rate and nature of degradation. The study also explores prompt engineering techniques, such as anti-slop and plan-first prompts, revealing their partial effectiveness in initial quality improvement but limited impact on long-term trends. These contributions collectively advance the understanding of AI code generation limitations and set the stage for future innovations in model robustness and sustainable development.
Novelty
This work is the first to systematically quantify long-term code degradation in AI agents through a dedicated benchmark, SlopCodeBench. Unlike prior evaluations focusing on correctness or short-term performance, this benchmark emphasizes the iterative extension process, capturing the gradual accumulation of complexity and redundancy. Its design, which isolates architectural decisions by only specifying external behavior, closely mimics real-world software development, making it highly relevant and practical. The introduction of structural erosion and verbosity as complementary metrics provides a nuanced understanding of code quality deterioration, offering new insights into the limitations of current models. The comparative analysis with human repositories further underscores the severity of the problem, highlighting the need for architectural and training innovations to address long-term sustainability.
Limitations
- 本研究主要在Python环境中进行评估，虽然框架设计为语言无关，但实际实验仅验证了Python，未来需扩展到其他编程语言以验证其普适性。模型在多轮迭代中的退化趋势可能受任务设计和模型特性影响，未能涵盖所有软件开发场景。
- 评估指标虽能反映代码复杂度和冗余，但未考虑代码的性能、可维护性和安全性等其他重要质量特性。在面对极端复杂或高风险任务时，模型表现仍未充分验证。
- 提示策略虽然在起始阶段改善了代码质量，但在长时间迭代中效果有限，未来需要结合模型架构优化和质量引导机制，才能更有效控制退化。
Future Work
未来工作应在多语言、多场景环境中验证SlopCodeBench的适用性，探索结合强化学习和自我监督的质量引导机制，提升模型在长时间任务中的稳定性。还需研究更智能的架构设计，结合动态质量反馈，减少代码退化。进一步扩展指标体系，涵盖性能、安全性和可维护性，构建更全面的评估体系。此外，推动模型自我修正和持续学习能力的发展，旨在实现自动编码系统的长效优化，推动自动化软件工程迈向更高水平。
AI Executive Summary
Software development is inherently an iterative process, involving continuous cycles of design, implementation, and refinement. With the advent of large language models (LLMs) such as GPT-4 and beyond, automated coding agents have gained prominence, promising to revolutionize software engineering by generating code autonomously. However, despite their impressive capabilities, these models often struggle with maintaining code quality over multiple iterations, especially when tasked with long-term, complex projects. Existing benchmarks primarily evaluate correctness in single-shot settings, neglecting how code evolves and deteriorates over time.
This gap motivated the development of SlopCodeBench, a novel benchmark designed to measure the long-term behavior of coding agents during iterative development. Unlike traditional benchmarks, which focus on initial correctness, SlopCodeBench emphasizes the progressive quality degradation of code as agents repeatedly extend their solutions. The benchmark comprises 36 diverse software development problems, each divided into multiple checkpoints—totaling 196—where the agent must iteratively improve or expand its codebase based solely on external behavioral specifications.
The core innovation of SlopCodeBench lies in its focus on two key metrics: structural erosion and verbosity. Structural erosion quantifies how complexity concentrates in functions that are already complex, measured via cyclomatic complexity and code mass. Verbosity captures redundant or bloated code patterns, identified through static analysis rules and clone detection. These metrics provide a nuanced view of code quality, revealing that most models, including GPT-5.5, achieve only around 14.8% solution rates, with code quality deteriorating steadily over checkpoints.
Empirical results demonstrate that no current model can fully solve the tasks end-to-end, and that code degradation—manifested as increased complexity and redundancy—is pervasive. The best-performing model, GPT-5.5, passes only 14.8% of checkpoints under strict criteria, while structural erosion and verbosity increase in over 75% of trajectories. Comparing agent-generated code with open-source repositories, the study finds that AI-produced code is 2.3× more verbose and 2.0× more eroded, highlighting significant challenges for long-term maintainability.
Prompt engineering strategies, such as anti-slop and plan-first prompts, can mitigate initial quality issues, reducing early-stage verbosity and erosion by over 30%. However, these improvements do not prevent ongoing degradation, which persists at an average rate of 1.3% per checkpoint. The findings underscore the importance of architectural improvements and quality-guided training to address the fundamental problem of long-term code deterioration.
Overall, SlopCodeBench provides a critical tool for evaluating and understanding the limitations of current AI coding systems. It exposes the risks of code bloating and complexity escalation during iterative development, emphasizing the need for more robust, sustainable approaches. Future research directions include enhancing model architectures, integrating dynamic quality feedback, and extending evaluations across multiple programming languages and real-world scenarios. This work lays the foundation for developing AI systems capable of producing maintainable, high-quality code over extended development cycles, ultimately advancing the goal of reliable automated software engineering.
Deep Dive
Abstract
Software development is iterative, yet agentic coding benchmarks hide design issues through their single-shot setup. Recent iterative benchmarks attempt to remedy this but heavily constrain an agent's design decision space, making it impossible to faithfully measure how their decisions shape future extensions. We introduce SlopCodeBench, a benchmark of 36 problems and 196 checkpoints where agents repeatedly extend their own solutions. Unlike prior iterative benchmarks, our evolving specifications demand architectural decisions but leave internal structure to the agent. We measure two forms of degradation: structural erosion (concentrated complexity) and verbosity (redundant code). Evaluating 15 coding agents across open and closed models, we find that no agent fully solves any problem end-to-end, and the best agent passes 14.8% of checkpoints. Quality degrades across checkpoints, with structural erosion rising in 77% of trajectories and verbosity in 75.5%. Compared to 473 open-source Python repositories, agent code is 2.3x more verbose and 2.0x more eroded, and the human repositories degrade less often and by smaller margins across their git histories. Explicit quality guidance reduces initial verbosity and erosion by up to a third, without affecting degradation rates. SlopCodeBench provides the first measurement of code degradation under iterative extension, revealing that agents pass checkpoints while producing code that erodes and bloats with each turn.
References (20)
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Carlos E. Jimenez, John Yang, Alexander Wettig et al.
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
Mike A. Merrill, Alexander G Shaw, Nicholas Carlini et al.
What is wrong with your code generated by large language models? An extensive study
Shihan Dou, Haoxiang Jia, Shenxi Wu et al.
Do code clones matter?
Elmar Jürgens, F. Deißenböck, B. Hummel et al.
Metrics for assessing a software system's maintainability
P. Oman, J. Hagemeister
The WyCash portfolio management system
Ward Cunningham
Software aging
D. Parnas
GPTCloneBench: A comprehensive benchmark of semantic clones and cross-language clones using GPT-3 model and SemanticCloneBench
A. Alam, Palash Ranjan Roy, Farouq Al-Omari et al.
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
John Yang, Carlos E. Jimenez, Alexander Wettig et al.
HumanEvo: An Evolution-Aware Benchmark for More Realistic Evaluation of Repository-Level Code Generation
Dewu Zheng, Yanlin Wang, Ensheng Shi et al.
RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems
Tianyang Liu, Canwen Xu, Julian McAuley
SWE-Bench+: Enhanced Coding Benchmark for LLMs
Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer et al.
Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarios
Zhi Chen, Lingxiao Jiang
Commit0: Library Generation from Scratch
Wenting Zhao, Nan Jiang, C. Lee et al.
LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks
Xin Zhou, M. Weyssow, Ratnadira Widyasari et al.
CodeIF-Bench: Evaluating Instruction-Following Capabilities of Large Language Models in Interactive Code Generation
Peiding Wang, Li Zhang, Fang Liu et al.
A Taxonomy of Inefficiencies in LLM-Generated Python Code
Altaf Allah Abbassi, L. D. Silva, Amin Nikanjam et al.
FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation
Wei Li, Xin Zhang, Zhongxin Guo et al.
ProjectEval: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generation
Kaiyuan Liu, Youcheng Pan, Jing Li et al.
MaintainCoder: Maintainable Code Generation Under Dynamic Requirements
Zhengren Wang, Rui Ling, Chufan Wang et al.
Cited By (15)
TRIM: Reducing AI-Generated CodeSlop via Agent Trajectory Minimization
StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turns
Repo0: Design-Driven Zero-to-All Code Generation
Persistent Recursive Worlds Enable Autonomous Software Evolution
SWE-Touch: Benchmarking Coding Agents When Users Touch the Code
LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation
Benchmarking the Residual: What Long-Horizon Evaluations Add Beyond Matched Short-Task Performance
EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environments
Is Agent Code Less Maintainable Than Human Code?
Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments
Does Code Cleanliness Affect Coding Agents? A Controlled Minimal-Pair Study
SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades
SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution
CodeTaste: Can LLMs Generate Human-Level Code Refactorings?
Towards Autonomous Software Development