SWE Context Bench: A Benchmark for Context Learning in Coding
Introduced SWE-ContextBench, a benchmark for evaluating context understanding and reuse in coding models, improving accuracy and efficiency significantly.
Key Findings
Methodology
This study constructs SWE-ContextBench based on real GitHub issues and pull request relationships, including 1100 base tasks and 376 related tasks. Manual analysis of dependencies and references extracts task sequences, forming shared context pools. Multiple retrieval strategies and context management methods are employed to evaluate model performance across different scenarios. Metrics such as accuracy variation, time, and token efficiency quantify the impact of context understanding. Experiments with models like Claude Code compare no-context, full-context, summaries, and optimal contexts, demonstrating that high-quality context summaries and retrieval significantly enhance task success rates while reducing runtime and token costs, especially on complex tasks.
Key Results
- High-quality summaries and retrieval notably improve task resolution. For example, GPT-5.3 Codex’s success rate increased from 22.60% to 23.94% with oracle summaries, reducing average token usage by about 27%. Models excelled particularly on difficult tasks, with runtime decreasing over 60%. Recursive relation expansion further enhanced task identification and context richness, confirming the importance of effective context management. Different retrieval strategies showed that optimal filtering and summarization are crucial for maximizing gains.
- Experimental results reveal that models leveraging high-quality context summaries outperform baseline setups. For instance, Claude Sonnet 4.5’s runtime dropped from 6.66 to 5.95 minutes with oracle summaries, and token costs decreased accordingly. The study also demonstrates that recursive expansion of task relations improves the identification of related tasks, enriching context pools and boosting learning efficiency. These findings underscore that managing and retrieving relevant context is vital for advancing code generation performance.
- The research validates that effective context understanding can lead to substantial improvements in accuracy and efficiency, particularly in complex software engineering tasks. The results suggest that models need not only to generate correct code but also to intelligently select and utilize past experiences, reducing redundant reasoning and computational costs. The introduced benchmark provides a comprehensive platform for further exploring these capabilities and optimizing context-aware models.
Significance
This work addresses a critical gap in evaluating models’ ability to learn from and reuse past experiences across related tasks, a key aspect in real-world software development. By establishing a systematic, multi-dimensional benchmark grounded in authentic development relationships, it enables precise measurement of context understanding and reuse. The findings highlight that effective context management can significantly boost model performance, paving the way for more intelligent, efficient coding assistants. This benchmark thus offers a vital tool for both academia and industry to push the boundaries of AI-driven software engineering, fostering models that learn continuously and adaptively in complex environments.
Technical Contribution
The paper introduces a novel evaluation framework combining real-world dependency analysis, multi-strategy retrieval, and recursive relation expansion to assess context learning in code models. It proposes a multi-metric evaluation system—accuracy variation, time, and token efficiency—providing a comprehensive understanding of model performance. The construction of task sequences based on actual GitHub relationships enhances realism, while the integration of summaries and optimal retrieval strategies advances the state-of-the-art in context-aware AI. These innovations collectively enable more precise and practical assessment of models’ ability to leverage past experience in multi-task settings.
Novelty
This is the first benchmark explicitly designed to evaluate cross-task context learning in software engineering, grounded in real dependency and reference relationships. Unlike existing long-context or memory benchmarks, SWE-ContextBench emphasizes the model’s ability to summarize, retrieve, and reuse prior experience across multiple related tasks, reflecting real development workflows. Its multi-strategy retrieval and recursive expansion mechanisms set it apart, providing a nuanced and realistic assessment platform that closely mimics industry scenarios.
Limitations
- The reliance on manual analysis for relationship extraction introduces subjectivity and scalability issues;自动关系识别技术尚未实现，未来需引入自动化工具。
- 模型在多依赖关系和复杂场景中的表现仍有限，特别是在错误筛选和上下文过滤方面存在不足，影响整体效果。
- 实验环境偏向理想化，未充分模拟工业环境中的噪声和多样性，未来需考虑更复杂的场景和多模态信息融合。
Future Work
未来将探索自动化关系识别与筛选技术，提升大规模场景下的适应性。结合多模态信息（如代码、文档、测试结果）优化上下文管理策略，推动工业级软件开发中的应用。此外，计划引入强化学习和元学习机制，增强模型的泛化和自主学习能力，推动上下文学习在实际软件工程中的落地。
AI Executive Summary
In modern software engineering, the ability of code agents to understand and reuse context is crucial for improving efficiency and accuracy. Traditional benchmarks mainly evaluate isolated performance, neglecting the importance of cross-task knowledge transfer. To address this gap, this paper introduces SWE-ContextBench, a comprehensive benchmark built on real-world GitHub relationships, including 1100 base tasks and 376 related tasks. These tasks are derived from actual issues and pull requests, capturing dependency and reference patterns that mirror real development workflows.
The benchmark constructs task sequences and shared context pools through manual analysis, enabling models to retrieve relevant past experiences when solving new problems. Multiple retrieval strategies—full, summary, and optimal—are tested across different models, such as Claude Code and GPT-5.3. Results demonstrate that high-quality summaries and accurate retrieval significantly boost task success rates, especially on complex tasks, while reducing runtime and token consumption. For instance, GPT-5.3 with oracle summaries achieves a resolution rate of 23.94%, compared to 22.60% without context, with token costs decreasing by nearly 27%. These findings highlight the importance of effective context management in real-world scenarios.
Furthermore, the study introduces recursive relation expansion, which captures second-order dependencies, enriching context pools and further improving performance. The results underscore that models must not only generate correct code but also learn when and how to leverage past experiences effectively. This benchmark provides a systematic, multi-dimensional framework for evaluating and advancing context-aware AI in software engineering. Future directions include automating relationship extraction, integrating multi-modal data, and applying reinforcement learning to foster autonomous, continual learning in code agents, ultimately transforming software development into a more intelligent and efficient process.
Deep Dive
Limitations & Outlook
What gaps remain?
Abstract
Large language models are increasingly used as coding agents for software engineering tasks. Current benchmarks mainly evaluate whether the agent can correctly solve the request or fix the bugs. They largely treat tasks as independent and do not assess whether agents can reuse previous experience across related problems. As a result, the efficiency gains from reusing the previous experience remains difficult to measure. We introduce SWE-ContextBench, a benchmark designed to explicitly evaluate context understanding and retrieval in coding agents. SWE-ContextBench consists of 1,100 base tasks with another 376 related tasks derived from real dependency and reference relationships among GitHub issues and pull requests. SWE-ContextBench groups base tasks and related tasks with shared context across 51 unique repositories and 9 programming languages. The benchmark evaluates how accurately and efficiently agents solve related issues when prior cases are available in context. Using SWE-ContextBench, we study the behavior of multiple coding agents across varying context reuse settings and retrieval strategies. Our results show that accurately summarized and retrieved previous experience can significantly improve resolution accuracy and reduce runtime and token cost, particularly on harder tasks. In contrast, unfiltered or incorrectly selected context provides limited or negative benefits. These findings highlight the importance of context management and retrieval accuracy, and position SWE-ContextBench as a principled benchmark for studying context learning in coding agents.