A Systematic Evaluation of Trajectory Data Curation for LoRA Fine-Tuning of Code Agents
This study systematically evaluates trajectory data filtering for LoRA fine-tuning of code agents, revealing data quantity outweighs quality at small scales.
Key Findings
Methodology
Using Qwen2.5-Coder-7B-Instruct, the authors propose a dual-axis quality scoring framework—Efficiency and Style—applied across 16 controlled experiments. They utilize cross-entropy (CE) loss on held-out trajectories as the primary metric, validated by first-action generation and ROUGE-L correlation (ρ = -1.00). The experiments compare strategies like random sampling, top-quality filtering, and ablations at scales of 500, 1000, and 2000 trajectories. The methodology emphasizes the importance of data scale, with detailed analysis of sub-metrics such as error-retry rate, step ratio, action diversity, and observation utilization, to understand their impact on model performance.
Key Results
- Doubling dataset size from 500 to 1000 trajectories results in approximately 12.7% reduction in CE loss, whereas the impact of quality filtering (TopQ vs. Random) remains under 1% at this scale. At 2000 trajectories, the gap widens to 3.6%, with p = 0.016, indicating scale-dependent effectiveness of filtering.
- Ablation shows that the error-retry rate (B2) is the dominant sub-metric, performing comparably to the full composite score, highlighting the importance of low-retry trajectories for better model learning.
- Proxy validation confirms CE loss correlates perfectly with first-action ROUGE-L (Spearman ρ = -1.00), supporting its use as a reliable performance indicator, especially at larger scales.
- Results demonstrate that data quantity has a more substantial effect than quality filtering at small scales, but as data grows, filtering strategies become increasingly beneficial, suggesting a scale-sensitive trade-off.
Significance
This research clarifies the relative importance of data quantity versus quality in fine-tuning open-source code agents. It establishes that increasing data volume yields larger performance gains than filtering quality alone at small scales, but the benefits of quality filtering emerge with scale. The multi-dimensional scoring framework and proxy-based evaluation protocol provide practical tools for optimizing data curation, addressing a key bottleneck in deploying efficient, capable code agents. These insights guide future efforts in data engineering and model training, especially under resource constraints, fostering more effective development of AI-powered software engineering tools.
Technical Contribution
The paper introduces a multi-dimensional, continuous quality scoring framework for trajectory data, integrating efficiency and style metrics. It innovatively employs CE loss as a performance proxy validated through first-action generation, enabling scalable, proxy-based evaluation. The systematic comparison of 16 filtering strategies across multiple scales reveals the scale-dependent impact of data quality, emphasizing the dominant role of low-retry trajectories. The approach advances the state-of-the-art in data curation for code agent fine-tuning, providing a quantifiable, scalable method to optimize training data selection and improve model capabilities efficiently.
Novelty
This work is the first to apply a multi-dimensional, continuous quality scoring system to multi-step agent trajectories, moving beyond binary success/failure filters. It uniquely validates CE loss as a proxy for downstream performance in this context and systematically analyzes the scale-dependent effects of data filtering. The comprehensive experimental design and ablation studies fill a critical gap in understanding how data quantity and quality interact in fine-tuning large language models for code tasks, offering new insights into scalable data curation strategies.
Limitations
- The experiments are limited to the Qwen2.5-Coder-7B-Instruct model and SWE-trajectory dataset, which may restrict the generalizability to other models or domains. Further validation across diverse architectures and data sources is needed.
- Proxy validation relies on a small number of checkpoints (n=4), which, while showing perfect correlation, may not fully capture the complexity of real-world performance variations, especially in more complex tasks.
- Scaling experiments are constrained by computational resources, with no validation at ≥5000 trajectories, leaving open questions about the crossover point and long-term scalability of the findings.
Future Work
Future research will extend validation to other models and datasets, explore trajectory-native quality metrics, and incorporate reinforcement learning for dynamic data filtering. Improving format-aware training to address JSON tool-call issues and scaling experiments at larger data volumes will be prioritized. Additionally, integrating these strategies into automated data curation pipelines could further enhance the efficiency and effectiveness of open-source code agent training, ultimately enabling more robust and capable AI-driven software development tools.
AI Executive Summary
The rapid advancement of large language models (LLMs) in software engineering has opened new avenues for autonomous code agents capable of complex multi-step reasoning. However, training such agents effectively remains a challenge, especially when relying on limited high-quality trajectory data. Traditional approaches emphasize increasing data volume or filtering for success, but the optimal balance between data quantity and quality is not well understood.
This study systematically investigates this trade-off by proposing a multi-dimensional, continuous quality scoring framework that evaluates trajectories along efficiency and style axes. Using the Qwen2.5-Coder-7B-Instruct model and SWE-trajectory dataset, the authors conduct 16 controlled experiments across different data scales (500, 1000, 2000 trajectories). They compare strategies such as random sampling, top-quality filtering, and ablations, with performance measured primarily via cross-entropy loss on held-out data, validated by first-action generation and ROUGE-L correlation.
Results reveal that increasing data size yields more substantial performance improvements than quality filtering at small scales. Specifically, doubling the dataset from 500 to 1000 trajectories reduces CE loss by approximately 12.7%, while filtering strategies contribute less than 1% at this scale. As data scales to 2000 trajectories, the benefit of quality filtering becomes more pronounced, with the gap widening to 3.6%, indicating a scale-dependent effect. Ablation studies identify low error-retry trajectories as the most influential sub-metric, emphasizing the importance of stable, low-retry data for effective learning.
These findings suggest that, in resource-constrained environments, prioritizing data volume may be more effective initially, but as datasets grow, quality filtering strategies can further enhance performance. The validated proxy metric (CE loss) offers a practical evaluation tool, especially when end-to-end resolve rates are statistically infeasible. Overall, this work provides a quantitative foundation for optimizing data curation in code agent fine-tuning, with implications for scalable, efficient AI software engineering solutions. Future directions include expanding validation across models and datasets, developing trajectory-native quality metrics, and integrating reinforcement learning to dynamically improve data selection, ultimately enabling more capable and resource-efficient AI-driven coding assistants.
Deep Dive
Abstract
Supervised fine-tuning (SFT) of open-weight LLMs on expert agent trajectories has emerged as a prominent approach to building capable code agents without reliance on proprietary models. A central yet underexplored question is how trajectory quality and quantity jointly shape model performance. We present a systematic empirical study of trajectory data filtering for LoRA fine-tuning of Qwen2.5-Coder-7B-Instruct on the SWE-trajectory dataset (67,074 trajectories, of which 32,161 are resolved). We propose a two-axis quality scoring framework -- Efficiency and Style -- and evaluate it through 16 controlled experiments spanning strategy, scale, and ablation analyses. Since 7B-scale models attain near-zero SWE-bench resolve rates, we adopt cross-entropy (CE) loss on held-out trajectories as the primary metric, validated via first-action generation: CE loss and ROUGE-L are perfectly rank-correlated (Spearman $ρ$ = -1.00), with limited-sample evidence supporting but not conclusively establishing this proxy. Our results reveal a scale-dependent quality-quantity trade-off: at small scales, doubling the dataset (500 to 1,000) yields ~12.7% CE-loss reduction whereas the TopQ-Random gap stays <1% (Mann-Whitney p > 0.10); at 2,000 trajectories this same gap widens to 3.6% (p = 0.016). Ablation further identifies error-retry rate as the dominant sub-dimension, performing comparably to the full composite ($Δ$ < 0.2%). Together, these findings establish trajectory-level quality scoring as a viable but scale-sensitive lever for code-agent SFT and offer a proxy-validated evaluation protocol for the regime where end-to-end resolve rate is statistically infeasible.