An Empirical Study of Harness Design for Coding Agents
Study improves coding agents' long-term performance via lightweight harness design; context management proves most effective.
Key Findings
Methodology
The paper introduces a lightweight coding harness with a fixed execution loop, varying three components: planning, action space, and context management. Evaluated using SWE-Bench Verified and Terminal-Bench 2.1 across 176 settings.
Key Results
- Context management is most effective under tight context-window budgets, preventing overflow failures, improving success rate by ~35%.
- Planning shifts from accuracy scaffold for weaker models to cost saver for stronger models, with little accuracy change.
- Predefined tools improve performance for bash-weak models, while bash-capable models achieve lower costs in command-line tasks.
Significance
This study reveals the independent utility of coding harness components, providing a modular framework for future design. Context management strategies significantly impact model and budget efficiency, offering solutions to prevent context overflow failures.
Technical Contribution
The paper advances SOTA methods by enabling independent evaluation of components through modular design, proposing new context management strategies and engineering possibilities.
Novelty
First to systematically compare the utility of planning, action space, and context management within a fixed execution loop, offering a modular evaluation approach.
Limitations
- Context management strategies' effectiveness diminishes under relaxed context-window budgets, reducing accuracy benefits.
- Planning component's cost-saving effect for strong models is limited.
Future Work
Future research could explore more combinations of context management strategies and their utility across different models and task types.
AI Executive Summary
Autonomous coding agents excel in software engineering tasks, but existing coding harnesses are typically evaluated as monolithic systems, obscuring the effectiveness of individual components. This paper proposes a lightweight coding harness, varying three components: planning, action space, and context management. Experiments show context management is most effective under tight context-window budgets, preventing overflow failures. Planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger models. Predefined tools improve performance for bash-weak models, while bash-capable models achieve lower costs in command-line tasks. These findings provide a modular framework for future harness design and reveal the independent utility of each component.
Deep Analysis
Background
As large language models are increasingly applied in software engineering tasks, coding harnesses become key to automation. Existing studies typically evaluate harnesses as complete systems, obscuring the effectiveness of individual components.
Core Problem
Existing coding harnesses obscure the effectiveness of individual components, hindering component-level comparisons and model performance optimization.
Innovation
Proposes a lightweight coding harness, varying planning, action space, and context management components, offering a modular evaluation approach.
Methodology
- �� Fixed execution loop framework
- �� Planning component maintains task progress
- �� Action space determines how model intentions become executable operations
- �� Context management strategy decides what interaction history remains available under a finite window
Experiments
Evaluated using SWE-Bench Verified and Terminal-Bench 2.1, assessing 176 settings, comparing five context management strategies and four context-window budgets.
Results
Context management is most effective under tight context-window budgets, preventing overflow failures. Planning shifts from accuracy scaffold for weaker models to cost saver for stronger models.
Applications
Can optimize coding agents' harness design, improving model performance in software engineering tasks.
Limitations & Outlook
Context management strategies' effectiveness diminishes under relaxed context-window budgets; planning component's cost-saving effect for strong models is limited.
Plain Language Accessible to non-experts
Imagine a kitchen where a chef needs to prepare multiple dishes in a limited space. Context management is like arranging ingredients and tools efficiently to ensure tasks are completed without space issues. Planning is like the recipe guiding each step. Action space is the chef's toolbox, determining how tools are used to complete the cooking.
ELI14 Explained like you're 14
Imagine playing a game where you need to complete multiple tasks in limited time. Context management is like managing your backpack in the game, ensuring you don't run out of space. Planning is like your task list, guiding you on how to complete each task. Action space is your skill tree, deciding how you use skills to complete tasks.
Glossary
Context Management
Determines what interaction history remains available under a finite window.
Used to prevent context overflow failures.
Planning
Component that maintains task progress.
Guides model operations.
Action Space
Determines how model intentions become executable operations.
Affects model execution efficiency.
SWE-Bench Verified
Benchmark testing repository-level issue resolution.
Used to assess model task success rate.
Terminal-Bench 2.1
Benchmark testing end-to-end terminal task completion.
Used to assess model task success rate.
Open Questions Unanswered questions from this research
- 1 How to enhance context management strategies' effectiveness under relaxed context-window budgets?
- 2 Planning component's cost-saving effect for strong models is limited; how to optimize?
Applications
Immediate Applications
Coding Optimization
Enhance coding agents' performance through modular design.
Long-term Vision
Intelligent Software Development
Achieve more intelligent software development processes, reducing human intervention.
Abstract
Coding harnesses shape how autonomous coding agents translate model capabilities into long-horizon software-engineering performance, yet existing work typically evaluates harnesses as monolithic systems, leaving the effectiveness of individual components unclear. To enable component-level comparisons, we study this question with a lightweight coding harness whose execution loop is fixed while three components are varied: planning, action space, and context management. Across four models evaluated on SWE-Bench Verified and Terminal-Bench 2.1, we evaluate 176 matched settings spanning five context-management strategies, four context-window budgets, and targeted ablations of planning and action space. We find that: (1) Context management becomes increasingly valuable as the context-window budget tightens, with most of its benefit coming from preventing context-overflow failures. (2) Staging rule-based elision before LLM-based summarization provides the strongest overall efficiency among the context-management strategies, whereas making elided content recoverable adds machinery that models rarely use and yields no accuracy gain. (3) Planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger models, with little change in accuracy. (4) Predefined tools improve performance for models with weaker bash proficiency, whereas bash-capable models can operate effectively with a bash-only interface and achieve substantially lower cost, especially on command-line-centric tasks. Trajectory-level analysis explains these effects: context management extends execution trajectories without substantially altering agent behavior, planning changes where trajectories stop, and the action space changes the granularity at which code is written. These findings inform model- and budget-aware harness design and provide a modular framework for evaluating future harness components.