Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation
Proposes PTA-IRT, integrating trajectory data to improve software agent benchmarking with 10% calibration, outperforming traditional IRT.
Key Findings
Methodology
PTA-IRT combines historical execution trajectories as privileged information, using structured summaries to encode long process logs. It employs a trajectory-aware 4PL model with Fisher information for task selection, and a LUPI-based teacher-student framework for ability estimation. The approach enhances low-budget performance recovery by leveraging process signals beyond binary outcomes, integrating deep neural encoders and statistical models for robust estimation.
Key Results
- On four SWE benchmarks, PTA-IRT with 10% calibration achieves an average MAE of 0.041, Kendall’s τ of 0.888, and Spearman’s ρ of 0.973, surpassing all baselines. In Lite and Verified datasets, it reduces MAE by over 20% and improves ranking metrics significantly. Ablation studies confirm the importance of trajectory summaries and information selection, with performance degrading when these components are removed.
- Budget sensitivity analysis shows that even at 5% calibration, τ reaches 0.768, demonstrating robustness. Compared to classical and neural IRT models, PTA-IRT maintains superior performance under minimal resource constraints, validating its practical utility.
Significance
This work addresses the high cost of evaluating complex software agents by leveraging process-level signals, enabling accurate performance estimation with minimal resource expenditure. It bridges the gap between static outcome-based metrics and dynamic process insights, offering a scalable solution for large-scale benchmarking. The framework advances the state-of-the-art in automated software evaluation, facilitating rapid development and deployment of intelligent coding agents, and has broad implications for AI-driven software engineering workflows.
Technical Contribution
Key innovations include the integration of structured trajectory summaries into a 4PL IRT model, the use of Fisher information for task stratification, and the application of LUPI for transferring process knowledge. The model combines deep neural encoders with statistical item response modeling, enabling nuanced understanding of task difficulty and agent ability. This hybrid approach improves estimation accuracy under limited budgets and extends IRT applicability to long-horizon, multi-step tasks in software engineering.
Novelty
This is the first work to incorporate detailed execution trajectories as privileged information within an IRT framework for software agent benchmarking. Unlike prior methods that rely solely on binary pass/fail responses, PTA-IRT leverages process signals to enhance task selection and ability inference, representing a significant paradigm shift. Its combination of structured summaries, Fisher information-based selection, and LUPI transfer learning sets a new standard for efficient, high-fidelity evaluation.
Limitations
- The effectiveness depends on the quality of trajectory summaries; noisy or incomplete logs can impair performance.
- In very complex or lengthy trajectories, summarization may omit critical information, affecting accuracy.
- Current validation is primarily in simulated or controlled environments; real-world industrial settings may introduce additional challenges such as system heterogeneity and real-time constraints.
Future Work
Future directions include developing multi-modal trajectory representations, improving summarization robustness, integrating reinforcement learning for dynamic task selection, and extending the framework to multi-task and multi-agent scenarios. Additionally, adapting the approach for real-time evaluation in industrial pipelines and exploring unsupervised trajectory encoding are promising avenues.
AI Executive Summary
As large language models become integral to software engineering, evaluating their capabilities at scale remains costly and complex. Traditional benchmarks require extensive code exploration, modification, and testing, leading to high resource consumption. This paper introduces PTA-IRT, a novel framework that leverages historical execution trajectories as privileged information to enhance performance estimation under limited calibration budgets. The core idea is to compress lengthy process logs into structured summaries, which capture key process signals such as explored context, edits, and problem-solving paths. These summaries inform a trajectory-aware 4PL item response model, enabling more accurate task difficulty assessment and agent ability inference.
The methodology involves three stages: parsing trajectories into structured summaries, selecting representative tasks based on Fisher information and difficulty stratification, and transferring process knowledge via a teacher-student LUPI paradigm. Extensive experiments on four SWE benchmarks demonstrate that PTA-IRT consistently outperforms classical and neural IRT models, achieving an average MAE of 0.041 and ranking metrics τ of 0.888 with only 10% calibration data. Even at 5% budgets, the model maintains high ranking agreement (τ=0.768), showing remarkable robustness.
The significance of this work lies in its ability to drastically reduce evaluation costs while maintaining high accuracy, facilitating large-scale benchmarking of software agents. It addresses a critical bottleneck in AI-assisted software development, enabling rapid iteration and deployment of intelligent tools. The approach’s generality suggests broad applicability across code generation, debugging, and repository reasoning tasks. Limitations include dependency on trajectory quality and challenges in real-world deployment, which future research aims to overcome. Overall, PTA-IRT marks a substantial step toward scalable, process-informed AI evaluation in software engineering.
Deep Analysis
Background
近年来，随着GPT-4、Codex等大模型在代码生成、缺陷检测等方面取得突破，软件工程自动化评估成为研究焦点。传统方法多依赖单轮测试或静态指标，成本高且难以反映复杂多步骤过程。项目响应理论（IRT）被引入，用于通过少量代表性任务估算模型能力，但多局限于二元结果，忽略了过程信息。深度IRT和PSN-IRT等模型引入神经网络增强表达，但仍未充分利用长轨迹中的过程信号。现有基准如SWE-bench面临高成本和低效率的问题，亟需更智能的评估策略。
Core Problem
软件工程代理在真实环境中的表现评估成本极高，尤其是在需要多步骤探索、修改和测试的复杂任务中。传统方法依赖大量任务执行，资源消耗巨大，难以实现大规模评估。现有的IRT方法只关注最终结果，忽略了过程中的关键线索，如探索路径、编辑尝试等。这导致在低预算条件下难以准确估算模型能力，影响模型的实际应用推广。如何在有限资源下，利用过程信息提升评估效率，成为亟待解决的核心问题。
Innovation
本研究提出轨迹感知的IRT（PTA-IRT），引入结构化轨迹摘要作为特权信息，结合信息熵和Fisher信息实现任务选择。创新点包括：1）利用长轨迹压缩提取过程信号，增强模型的解释能力；2）引入轨迹感知的4参数逻辑模型，提升任务难度估计的准确性；3）采用LUPI机制，将轨迹信息迁移到能力估计中，显著改善低预算下的性能恢复。该框架融合深度学习与统计模型，突破传统二元响应限制，提升评估效率。
Methodology
- �� 轨迹解析：将长文本轨迹转化为结构化摘要，包括任务目标、探索内容、编辑路径和解决方案路径。• 任务选择：基于轨迹感知的4PL模型，利用Fisher信息和信息熵对任务进行难度和信息量评估，形成代表性子集。• 能力估计：在训练阶段，利用轨迹摘要训练教师模型，将过程信号迁移到学生模型；在测试阶段，只用有限任务的响应估算模型整体能力。• 结构化摘要：采用深度学习模型（如MiniLM）提取摘要特征，增强轨迹信息的表达。• 任务筛选：通过信息量最大化策略，确保子集覆盖不同难度区间，提升估算精度。
Experiments
在四个SWE基准（Lite、Verified、Full、Pro）上进行验证，比较多种IRT变体和AutoJudger。采用10%预算进行任务选择，评估指标包括MAE、Kendall’s τ和Spearman’s ρ。通过交叉验证确保结果稳健，调优模型参数。 Ablation研究验证轨迹摘要和信息选择机制的重要性。结果显示，PTA-IRT在低预算下优于所有对比方法，尤其在排名一致性和分数恢复方面表现突出。
Results
在四个基准中，PTA-IRT平均MAE为0.041，排名指标τ达0.888，明显优于传统IRT（MAE约0.171，τ约0.528）和深度IRT。5%预算已实现τ值0.768，显示出极强鲁棒性。消融实验表明，去除轨迹摘要或信息选择机制会导致性能下降约30%。此外，轨迹摘要的引入显著改善了模型对复杂任务的适应能力，验证了其在实际场景中的潜力。
Applications
该方法适用于大规模软件工程自动化评估、代码生成模型能力排序、缺陷检测系统性能监控等场景。只需少量代表性任务，即可准确估算模型整体表现，节省大量计算资源。未来可结合工业环境中的持续集成流程，实现实时性能监控和模型优化，推动自动化开发工具的普及。
Limitations & Outlook
模型依赖轨迹摘要的质量，若轨迹信息噪声大或不完整，可能影响估算效果。复杂或长时间轨迹的摘要可能丢失关键信息，影响能力推断。当前方法主要在模拟环境验证，实际工业场景中还需考虑系统复杂性、实时性和多模态信息融合等因素。未来需优化轨迹解析和摘要生成算法，提升鲁棒性和适应性。
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂里每个工人（模型）都要完成一项任务，比如组装产品。传统评估就像只看工人最后是否成功（成功或失败），但实际上工人在过程中会经历很多步骤，比如试错、调整、学习。现在，研究者发现，如果能记录工人在每一步的行动（轨迹），就像看工人在工厂里的每个动作一样，就能更好地判断他们的能力。这个方法就像教练不仅看比赛结果，还观察比赛中的每个动作，帮助更准确地评估工人的水平。这样，即使只观察一部分比赛，也能知道工人整体水平，节省时间和成本。
ELI14 Explained like you're 14
想象你在玩一款游戏，你的朋友想知道你到底多厉害，但又不想看你每次玩完整个游戏。于是，你们决定只看你玩了哪个关卡、怎么过的、用了哪些技巧。通过这些细节，朋友可以大概知道你整体的水平，而不用看你每次打完所有关卡。这个研究就是用类似的方法，只看软件模型在一些任务中的表现，但还结合了它在过程中做了什么、试了哪些方法，来更准确地估算它的整体能力。这样既省时间，又能得到更靠谱的评估结果。
Abstract
Evaluating software engineering agents on realistic benchmarks is costly, since each task may require multi-step code exploration, modification, and test execution. Existing efficient evaluation methods select representative subsets to estimate full-benchmark performance, but are largely result-only: they fit historical pass/fail response matrices or static task semantics, discarding how agents solve problems. We propose PTA-IRT, a Privileged Trajectory-Aware Item Response Theory framework that fuses process and outcome signals. Historical execution trajectories supply process-level evidence beyond pass/fail, such as explored context, attempted edits, and solving paths, which PTA-IRT uses as privileged information for calibration subset selection and ability estimation. Under low calibration budgets, PTA-IRT consistently outperforms prior IRT baselines on score and ranking recovery across four SWE benchmarks. Code and data are publicly available at https://github.com/DeepSoftwareAnalytics/PTA-IRT.