GptGet

Immersion in the GitHub Universe: Scaling Coding Agents to Mastery

TL;DR

Proposes ScaleSWE, an automated multi-agent system for large-scale construction of high-quality SWE datasets, producing 100k verified instances.

cs.SE 🔴 Advanced 2026-02-10 49 views
Jiale Zhao Guoxin Chen Fanzhe Meng Minghao Li Jie Chen Hui Xu Yongshuai Sun Wayne Xin Zhao Ruihua Song Yuan Zhang Peng Wang Cheng Chen Jirong Wen Kai Jia
AI Reader Arxiv Page Download PDF
software engineering large-scale data multi-agent automation LLMs

Key Findings

Methodology

This work develops a sandboxed multi-agent framework comprising environment builder (EBA), test creator (UCA), and problem writer (PSWA). These agents coordinate to process 5,200 repositories and 6 million PRs, automating environment setup, test generation, and task description synthesis. Deep learning models like DeepSeek-V3.2 and Gemini3-Pro are employed for trajectory distillation, fine-tuning Qwen-30B-A3B-Instruct to create ScaleSWE Agent. The system ensures high data quality through rule-based filtering and expert validation, resulting in a dataset with 100,000 verified instances, surpassing existing benchmarks in diversity and complexity.

Key Results

  • The dataset contains 100,000 verified instances with diverse repositories and complex tasks, outperforming prior datasets. From 25,000 instances, 71,498 high-quality trajectories were distilled, enabling the fine-tuned ScaleSWE Agent to achieve a 64% success rate on SWE-Bench, nearly triple the baseline.
  • Automated environment configuration, test synthesis, and task description significantly reduce manual effort, enhancing data authenticity and variety. The model's performance validates the dataset's utility.
  • Experimental results demonstrate that environment reuse and trajectory distillation greatly improve construction efficiency and model robustness, providing a new paradigm for large-scale automated software data collection.

Significance

This research addresses the critical bottleneck of data scarcity in software engineering AI. By enabling scalable, automated construction of high-quality datasets, it facilitates the deployment of large language models in real-world coding tasks. The approach overcomes challenges like environment heterogeneity, test generation difficulty, and task ambiguity, laying a foundation for future industrial applications. The generated datasets not only boost model performance but also promote the development of intelligent, automated software engineering tools, accelerating the industry’s move toward automation and AI-assisted development.

Technical Contribution

The paper introduces a novel multi-agent framework integrating environment inference, test synthesis, and task description generation, all driven by deep learning. It innovates in environment reuse strategies and trajectory distillation, enabling scalable, high-fidelity data construction from open-source repositories. The system's modular design allows flexible adaptation and extension, providing a new blueprint for automated large-scale dataset creation in software engineering. The fine-tuning of Qwen-30B-A3B-Instruct with distilled trajectories demonstrates the practical impact of this data pipeline, significantly improving task-solving capabilities.

Novelty

This is the first comprehensive implementation of a multi-agent, fully automated pipeline for large-scale SWE data construction, covering environment setup, test generation, and task synthesis. Unlike prior work relying on manual curation or rule-based methods, this approach leverages deep learning for environment inference and test synthesis, ensuring realism and diversity. The integration of trajectory distillation for model fine-tuning further distinguishes it, providing a scalable, end-to-end solution for real-world software data generation.

Limitations

  • The current system mainly targets Python repositories; adapting to other languages requires additional dependency inference and environment configuration strategies. Extreme dependency complexity may still cause failures.
  • Test case generation, while automated, may occasionally produce false positives or negatives, affecting validation accuracy. Enhanced verification mechanisms are needed.
  • Trajectory distillation is computationally intensive, requiring substantial resources. Future work should focus on optimizing efficiency and reducing costs.

Future Work

Future directions include expanding multi-language support, improving environment robustness, and integrating active learning to optimize data collection. Developing more efficient trajectory distillation algorithms and deploying the system in industrial settings are also planned. Additionally, exploring end-to-end automation for entire software development workflows, from code analysis to deployment, will further advance AI-assisted software engineering.

AI Executive Summary

In recent years, large language models (LLMs) have demonstrated remarkable capabilities in code understanding and generation, yet their success heavily depends on the availability of high-quality, large-scale training data. Traditional data collection methods, often manual or rule-based, are labor-intensive and limited in diversity, hindering the development of truly robust AI code assistants.

This paper introduces ScaleSWE, a pioneering framework that automates the construction of extensive, realistic software engineering datasets using a multi-agent system. The core idea is to leverage three specialized agents—environment builder (EBA), test creator (UCA), and problem writer (PSWA)—to process millions of pull requests from thousands of open-source repositories. EBA automatically configures isolated Docker environments tailored to each repository, ensuring reproducibility. UCA synthesizes comprehensive unit tests, including Fail-to-Pass and Pass-to-Pass cases, by analyzing PR metadata and code context. PSWA then generates formal, self-contained task descriptions grounded in the test suites.

The system's pipeline processed over 6 million PRs, resulting in 100,000 verified instances, making it the largest real-world SWE dataset to date. The dataset reflects diverse repository origins and complex tasks, capturing real-world software evolution intricacies. To demonstrate its utility, the authors distilled 71,498 high-quality trajectories from 25,000 instances using DeepSeek-V3.2, then fine-tuned Qwen-30B-A3B-Instruct. The resulting ScaleSWE Agent achieved a 64% success rate on SWE-Bench, nearly tripling the baseline performance.

This work marks a significant advance in automated data construction for software AI, offering a scalable, reproducible approach that bridges the gap between raw open-source artifacts and high-quality training data. It paves the way for more robust, industry-ready AI tools capable of understanding and repairing complex code, ultimately accelerating the integration of AI into real-world software development. Future efforts will focus on multi-language support, efficiency improvements, and industrial deployment, aiming to transform software engineering into a more automated, intelligent domain.

Deep Analysis

Background

Software engineering (SE) has evolved rapidly with the advent of large language models (LLMs), which have demonstrated promising results in code synthesis, comprehension, and repair tasks. Early datasets like CodeSearchNet and DeepCode relied heavily on manual annotations or limited rule-based data, constraining their scale and realism. Recent efforts such as SWE-Gym and SWE-smith attempted to leverage open-source repositories, but faced challenges in ensuring data authenticity, diversity, and environment reproducibility. The complexity of real-world software, with dependencies, build environments, and intricate test requirements, remains a bottleneck. As LLMs continue to improve, the demand for large, high-quality, diverse datasets that mirror real-world software development grows. Overcoming the environmental heterogeneity, test generation difficulty, and task ambiguity is crucial for advancing AI-assisted software engineering.

Core Problem

Current data collection methods struggle with automating environment setup, generating comprehensive unit tests, and creating precise task descriptions. Environment configuration is often fragile due to dependency complexity, requiring manual intervention. Many repositories lack sufficient tests, making validation difficult. Additionally, task descriptions are frequently incomplete or biased, leading to inconsistent training data. These issues limit dataset scale, diversity, and realism, impeding the development of robust models capable of handling real-world software tasks. Addressing these bottlenecks with an automated, scalable pipeline is essential for progress in AI-driven software engineering.

Innovation

This work introduces a multi-agent, automated pipeline that integrates environment inference, test synthesis, and task description generation. Key innovations include:

  • �� Environment Builder Agent (EBA): uses deep learning to analyze source code and dependencies, automatically configuring Docker environments, enabling scalable environment setup.
  • �� Test Creator Agent (UCA): synthesizes Fail-to-Pass and Pass-to-Pass tests from PR metadata and code context, ensuring validation robustness.
  • �� Problem Writer Agent (PSWA): generates formal, self-contained task descriptions aligned with test suites, avoiding solution leakage.

These components work collaboratively, leveraging deep learning models like DeepSeek-V3.2 and Gemini3-Pro, to produce high-fidelity, diverse datasets from millions of open-source PRs, significantly reducing manual effort and bias.

Methodology

  • �� 数据采集：筛选PyPI前1.5万下载量最高仓库和GitHub满足特定条件的项目，合计约2.3万仓库。
  • �� PR筛选：利用LLM自动过滤低质量PR，得到600万PR样本。
  • �� 环境配置：环境代理（EBA）分析源码依赖，利用深度学习推理生成Docker环境，采样最多10个PR构建环境，环境重用策略提升效率。
  • �� 测试生成：测试代理（UCA）分析PR元数据和源码，自动合成Fail-to-Pass和Pass-to-Pass测试用例，利用沙箱环境（Dfinal）动态验证。
  • �� 任务描述：任务代理（PSWA）结合PR元数据和测试内容，自动生成无泄露的任务描述。
  • �� 数据验证：采用规则过滤和专家人工审核确保数据质量，最终生成10万验证实例。
  • �� 轨迹蒸馏：利用DeepSeek-V3.2从25k实例中蒸馏7.1万高质量轨迹，用于微调模型。
  • �� 模型训练：微调Qwen-30B-A3B-Instruct，验证在SWE-bench上的性能提升。

Experiments

采用SWE-bench验证模型性能，比较基础模型与微调模型的解决率。训练参数包括学习率1e-5，批次128，最大上下文长度131072。模型在推理时扩展到262144字节以处理大输入。通过多轮轨迹采样和验证，确保数据的高质量。对比SWE-Gym、SWE-smith等数据集，验证ScaleSWE在多样性和真实性方面的优势。实验证明，微调后模型在SWE-bench验证中解决率由22%提升至64%，显示出数据质量和规模的显著影响。

Results

数据集规模达10万实例，涵盖丰富的仓库多样性和复杂任务，优于现有公开数据集。通过轨迹蒸馏产生7.1万高质量轨迹，训练出的ScaleSWE Agent在SWE-Bench验证中解决率达64%，较基础模型提升近三倍。这证明了自动化数据采集和轨迹蒸馏的有效性。实验还显示环境重用策略极大提升了构建效率，模型在实际任务中的表现优异，验证了数据集的实用价值。

Applications

该方法可广泛应用于自动化软件测试、缺陷修复和代码理解等场景。企业可以利用此系统快速构建定制化的训练数据，提升AI辅助开发工具的性能。未来结合工业界的持续集成环境，推动自动化研发流程，降低人力成本，提升软件质量。

Limitations & Outlook

系统目前主要针对Python仓库，跨语言迁移仍需适配。环境配置在极端复杂场景下可能失败，影响数据完整性。生成的测试用例偶尔存在误判，验证准确性需加强。轨迹蒸馏成本较高，未来需优化算法以降低训练资源需求。

Plain Language Accessible to non-experts

想象你在一个大型厨房里，厨师们需要准备各种菜肴。每个厨师负责不同的任务：一个负责准备食材（环境配置），一个负责试味（测试生成），还有一个负责写菜谱（任务描述）。他们通过合作，自动化地准备出丰富多样的菜肴（软件任务实例），而且每道菜都经过严格检验（验证测试）。这样，厨房可以快速生产出大量高质量的菜肴，满足不同客人的需求。这个过程就像本文中的多智能体系统，自动搭建环境、生成测试、描述任务，最终帮助AI学习像人一样理解和修复代码。整个系统节省了大量人工劳动，还能保证菜肴（数据）丰富多样，真实可靠。

ELI14 Explained like you're 14

想象你在学校的科学实验室里，有很多不同的实验需要准备。有的实验需要特定的设备（环境配置），有的需要设计测试（测试生成），还有的需要写出实验说明（任务描述）。以前，老师都得一个一个手工准备，非常麻烦。现在，有一群机器人助手：一个帮忙搭建实验环境，一个帮忙设计测试题，一个帮忙写实验说明。这些机器人合作无间，自动完成所有准备工作，快速生产出很多完整的实验方案（软件任务实例）。每个方案都经过严格检查，确保没有错误。这样，学生们就可以用这些方案来学习和练习，而老师也省了很多时间。这就像本文中的多智能体系统，自动搭建软件开发的“实验环境”，生成“测试题”和“任务描述”，帮助AI更好地理解和修复代码。整个过程既高效又可靠，节省了大量人工劳动，还能产生丰富多样的学习材料。

Glossary

沙箱环境 (Sandbox Environment)

一种隔离的运行空间，用于安全测试和验证代码，确保不会影响其他系统。

用于环境配置代理（EBA）自动搭建隔离环境。

Fail-to-Pass (F2P) 测试

最初测试失败，修复后应通过的测试，用于验证缺陷修复效果。

由测试代理（UCA）生成，用于验证代码修正的有效性。

轨迹蒸馏 (Trajectory Distillation)

从自动交互轨迹中提取高质量数据，用于微调大模型，提高其任务能力。

用于从自动生成的交互中训练更强的代码修复模型。

多智能体系统 (Multi-Agent System)

由多个协作的智能体组成的系统，各自执行特定任务，共同完成复杂目标。

本文设计的自动化软件数据构建框架基础。

环境代理 (EBA)

自动分析源码仓库，配置隔离环境的智能体。

实现环境的自动配置和重用。

Open Questions Unanswered questions from this research

  • 1 如何进一步提升环境配置的鲁棒性，支持更多编程语言和复杂依赖。
  • 2 测试用例生成的准确性和效率仍有提升空间，特别是在极端场景下的验证机制。
  • 3 大规模轨迹蒸馏的成本较高，未来需优化算法以降低训练资源需求。

Applications

Immediate Applications

自动化软件缺陷修复

企业可以利用该系统快速生成训练数据，提升AI在代码修复中的表现，缩短开发周期。

智能代码理解工具

为代码分析和理解提供丰富的训练样本，增强AI的推理和诊断能力。

Long-term Vision

工业级自动化研发平台

结合自动数据采集和模型微调，推动软件开发流程的全面自动化，降低人力成本，提升软件质量。

Abstract

Achieving mastery in real world software engineering tasks is fundamentally bottlenecked by the scarcity of large scale, high quality training data. Scaling such data has been limited by the complexity of environment setup, unit test generation, and problem statement curation. In this paper, we propose ScaleSWE, an automated, sandboxed multi agent workflow designed to construct high quality SWE data at scale. The system coordinates three specialized agents for environment setup, test creation, and problem description synthesis to process 6 million pull requests across 5200 repositories, producing Scale SWE Data: 100k verified SWE instances, the largest such dataset to date. It substantially surpasses existing real world datasets in repository diversity and reflects realistic task complexity. We further demonstrate the dataset utility for training by distilling 71498 high quality trajectories and finetuning Qwen30BA3BInstruct to produce ScaleSWE Agent. Our agent achieves a 64 resolve rate on SWE Bench Verified a nearly three fold improvement over the base model. ScaleSWE provides a scalable, reproducible approach for data construction to advance LLM based software engineering. Scale SWE will be publicly available.

cs.SE