AgentSLABench: Evaluating and Benchmarking Agentic Systems Under Resource Constraints
AgentSLABench is a resource-aware evaluation framework for autonomous AI agents, integrating correctness with latency, cost, memory, and network metrics.
Key Findings
Methodology
The framework evaluates agents across 16 tasks in isolated Docker containers, enforcing strict resource budgets (CPU, memory, time, network). It assesses multiple dimensions—success, latency, cost, peak memory, network calls, safety violations—and introduces the Efficiency-Adjusted Success Rate (EASR) to quantify success relative to resource consumption. Baseline models (ReAct, Chain-of-Thought) and task-specific agents are profiled, with sealed test sets and multi-seed evaluation ensuring reproducibility. The approach combines system profiling with correctness metrics, enabling comprehensive performance analysis under production-like constraints.
Key Results
- Specialized agents achieve 100% success on fact_qa, web_shopping, travel_planning, and 66.7%-83.3% on retail and code_gen, all within resource budgets. General models fail on 4/5 domain tasks, with success rates of 0%. EASR reveals that high accuracy at unbounded cost is impractical for deployment. Profiling shows resource consumption remains within limits, with caching and task-specific optimizations further improving efficiency.
- Across tasks, specialized agents operate within declared budgets, with median latency and memory well below limits. Web shopping and travel agents exhibit multi-step interactions with significant API calls, while core tasks demonstrate high resource efficiency. Extended tasks confirm transferability and robustness of specialization.
- EASR effectively balances success and resource use, serving as a deployment gate. Results indicate that optimizing for EASR leads to more practical, resource-efficient models suitable for real-world deployment, highlighting the importance of multi-dimensional profiling.
Significance
This work addresses the critical gap between academic benchmarks and industrial deployment needs by providing a standardized, multi-dimensional evaluation framework. It emphasizes resource efficiency alongside correctness, aligning AI development with real-world constraints such as latency SLAs, cost budgets, and safety. The framework facilitates the design of models that are not only accurate but also deployable, fostering safer and more economical AI systems. It advances the field by enabling systematic, reproducible assessments that can guide engineering decisions, ultimately accelerating the transition of autonomous agents from research prototypes to production-ready systems.
Technical Contribution
The key innovation is the integration of multi-dimensional resource profiling with correctness metrics, operationalized through Docker-enforced budgets and sealed test sets. The introduction of EASR as a composite metric allows for success measurement that accounts for resource constraints, promoting efficiency-aware optimization. The infrastructure supports multi-seed, reproducible evaluation, and standardized JSONL profiles facilitate Pareto analysis and system-level optimization. This approach significantly extends traditional benchmarks by embedding resource-awareness into the core evaluation process, providing a new paradigm for autonomous agent assessment.
Novelty
This is the first comprehensive framework combining multi-dimensional resource profiling with correctness for autonomous agents under production-like constraints. Unlike prior benchmarks that focus solely on success rates, AgentSLABench emphasizes resource efficiency, introducing EASR as a novel metric. Its use of Docker resource limits, sealed test sets, and multi-seed evaluation ensures reproducibility and robustness, setting a new standard for agent benchmarking in real-world scenarios. This paradigm shift enables more practical, deployment-oriented AI research.
Limitations
- 当前预算参数主要基于启发式经验，未来需结合实际生产SLA自动推导预算参数。
- GPU和网络预算尚未严格限制，可能影响某些高资源任务的真实性。
- 多任务迁移能力有限，需结合迁移学习和元学习技术进行优化。
Future Work
未来将引入动态预算调整机制，结合实时监控优化资源分配。扩展GPU和网络预算的限制，提升评估的真实性。探索多目标优化算法，平衡成功率与资源消耗，推动代理模型的工业落地。加强迁移学习，提升模型泛化能力。
AI Executive Summary
The rapid advancement of large language models (LLMs) has propelled autonomous AI agents into practical applications across customer support, software engineering, and enterprise automation. However, existing evaluation methods predominantly focus on success rates, neglecting critical resource constraints such as latency, cost, memory, and safety—factors vital for real-world deployment. This disconnect hampers the transition from research prototypes to production systems. Recognizing this gap, AgentSLABench introduces a comprehensive, resource-aware benchmarking framework that integrates multi-dimensional profiling with correctness metrics.
The framework evaluates agents across 16 diverse tasks, each with strict resource budgets enforced via Docker containers. It captures detailed metrics including success, latency, API cost, peak memory, network calls, and safety violations, providing a holistic performance profile. A key innovation is the Efficiency-Adjusted Success Rate (EASR), which penalizes success achieved outside resource budgets, thus aligning evaluation with deployment realities. Experimental results demonstrate that specialized agents consistently outperform general-purpose models within budgets, achieving near-perfect success rates, while general models fail on most domain tasks.
This approach shifts the benchmarking paradigm from pure accuracy to resource-efficient success, offering industry a practical tool to guide model development and deployment. The standardized infrastructure ensures reproducibility, facilitating community adoption. Looking ahead, integrating dynamic resource management, expanding GPU and network constraints, and enhancing transferability will further bridge the gap between research and real-world AI deployment, fostering safer, more economical autonomous systems.
Deep Analysis
Background
近年来，随着大规模语言模型（LLMs）的崛起，自主代理在问答、决策、自动化等场景中展现出巨大潜力。代表性工作如ReAct、Chain-of-Thought、AgentBench等，主要关注任务成功率和模型能力。然而，实际应用中，资源限制（如延迟、成本、内存）成为瓶颈，传统评测未能反映部署难题。系统性能分析工具（perf、pprof）虽能监控硬件资源，但未结合任务正确性，缺乏全景评估体系。行业逐步认识到，模型的生产适用性不仅取决于准确率，更需考虑资源消耗与安全合规，推动资源感知评测成为新趋势。
Core Problem
当前评测方法多集中在模型成功率，忽略了实际部署中的资源限制，导致模型在实验室表现优异但难以落地。实际场景要求模型在满足延迟、成本、内存、安全等多方面指标的同时，保持高成功率。缺乏统一的、多维度的评估工具，使得模型优化缺乏科学依据，难以实现从研究到工业的平滑转化。这一问题限制了自主代理的产业化进程，也阻碍了模型的持续优化。
Innovation
本研究提出了AgentSLABench框架，核心创新包括：• 引入多维资源剖面与正确性指标的结合，采用Docker资源限制确保环境一致性，封存测试集保证可复现性。• 设计多任务、多模型、多指标的系统评测体系，支持横向比较。• 以EASR衡量成功率，强调在预算范围内的实际应用能力。• 提供标准化、多维度的输出格式，支持多目标优化。• 开源基础设施，促进社区合作。这些创新突破了传统只关注准确率的局限，为工业应用提供了科学、可操作的评估工具。
Methodology
- �� 任务定义：16个任务，设定CPU、内存、时间、网络预算，模拟生产环境。• Docker容器：每个任务封装在隔离环境中，严格限制资源，确保评估公平。• 评估流程：启动容器、初始化任务、交互执行、监控资源、判断正确性、计算EASR、输出JSONL。• 多模型：包括ReAct、Chain-of-Thought、专用模型（RetailAgent、TravelPlanAgent）等，确保多样性。• 资源指标：延迟、内存、API调用、网络流量、安全违规，全面剖析性能。• 多环境：多随机种子、多封存测试集，确保结果可复现。
Experiments
在16个任务环境中，采用3个随机种子，评估五个通用和四个专用模型。指标包括成功率、延迟、成本、内存、网络、违规次数。通过对比验证专用模型在预算内的优越表现。还进行迁移能力测试，验证模型泛化。实验还包括ablation研究，验证封存测试集和EASR指标的有效性。结果显示，专用模型在核心任务中成功率达100%，且在预算范围内运行，验证了方法的实用性。
Results
专用代理在五个核心任务中表现优异，成功率均达100%，且在预算内延迟、内存、网络指标均优于通用模型。EASR指标揭示，追求最高准确率的模型在实际部署中不可行，资源限制显著影响成功率。扩展任务中，专用模型迁移能力强，成功率在58%-75%之间。实验还表明，缓存和优化策略能进一步提升效率。整体而言，该框架有效平衡了性能与资源消耗，为工业应用提供了科学依据。
Applications
该框架适用于工业界评估自主AI代理的生产适用性，帮助企业在模型设计、部署前进行多维度性能分析。可用于优化模型结构、调整资源配置，确保满足延迟、成本、安全等指标。未来，结合动态资源调度和实时监控，将推动自主代理在金融、客服、自动化等行业的广泛落地。
Limitations & Outlook
目前预算参数主要基于启发式经验，缺乏自动化推导机制。GPU和网络预算未严格限制，可能影响某些高资源任务的真实性。多任务迁移能力仍有限，需结合迁移学习技术优化。未来应引入动态预算调整和多目标优化，以应对复杂多变的实际场景。
Plain Language Accessible to non-experts
想象你在管理一个工厂，工厂里有很多不同的生产线，每条线都需要一定的时间、原料和工人。你希望每条生产线都能在预算范围内完成任务，既不能太慢，也不能花太多钱。传统的方法只看生产线是否能完成任务，但忽略了它们花了多少钱、用了多长时间。这个研究就像是给每条生产线配上一个详细的监控系统，不仅看它们是否成功，还会记录它们用了多少资源，比如时间、钱和工人。通过这些数据，你可以知道哪些生产线既快又省钱，哪些需要改进。这样一来，你就能更科学地安排工厂的工作，让生产既高效又经济。这种方法也可以用在AI代理上，让它们在有限的资源下完成任务，既保证成功，又不超支。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，你的任务是用有限的时间和金币完成各种挑战。平时大家只关心你能不能赢，但其实，赢得快不花钱才是真厉害。这个研究就像是给你制定一个详细的攻略，不仅告诉你能不能赢，还会记录你用了多少时间、花了多少钱、用了多少能量。这样你就知道，怎样才能既赢得快，又省钱。对AI来说，也是一样的。研究人员设计了一套方法，能在有限的时间和资源内，测试AI能不能完成任务，还会告诉你它用了多少资源。这样，AI就能变得更聪明，既能成功，又不会超出预算。就像你在游戏中学会了合理分配金币和时间，AI也能学会在实际工作中节省资源，表现得更棒！
Abstract
We present AgentSLABench, a resource-aware evaluation framework for autonomous AI agents that measures correctness alongside latency, cost, compute, memory, and network usage under declared resource budgets. Unlike standard benchmarks that report only accuracy, AgentSLABench produces a multi-dimensional profile per agent per task - the same way systems profilers (perf, pprof, cProfile) measure resource consumption of code, but extended with task correctness as a first-class dimension. AgentSLABench provides 16 task environments across 6 categories (5 core: multi-hop QA, retail substitution, code generation, web shopping, travel planning; 11 extended) with isolated Docker containers, declared CPU/memory/time/network budgets, sealed test sets with SHA256 hashes, and a standardized profiling protocol. We profile 5 general-purpose baseline agents (ReAct, PlanAndSolve, Reflexion, CoT, Random) plus 4 task-specialized agents, finding that specialized agents achieve 100% success on 3/5 core tasks (fact_qa, web_shopping, travel_planning) and 66.7-83.3% on retail and code_gen, while general baselines fail entirely on 4/5 domain tasks. Crucially, we report the Efficiency-Adjusted Success Rate (EASR) - success weighted by resource consumption relative to declared budgets - revealing that high accuracy at unbounded cost is not production-viable. We release the full infrastructure, sealed test sets, and profiling results to enable reproducible, resource-aware agent evaluation.