VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?
VibeServe employs multi-agent loops to automatically generate bespoke LLM serving systems, matching or exceeding hand-tuned performance.
Key Findings
Methodology
VibeServe utilizes a hierarchical framework with an outer planning loop and an inner implementation loop. The outer loop leverages git history and optimization records to select design targets, while the inner loop employs multiple specialized agents—Implementer, Accuracy Judge, Performance Evaluator—to collaboratively generate, validate, and measure candidate systems. Communication is facilitated via a Model Context Protocol (MCP), and a skills library provides targeted knowledge for model architectures, hardware platforms, and optimization techniques. The process iterates over multiple cycles, progressively refining system design to meet correctness and performance goals across diverse scenarios.
Key Results
- In standard deployment (Llama-3.1-8B-Instruct on H100), VibeServe achieves near parity with vLLM in throughput and latency, with less than 5% TTFT deviation, and surpasses SGLang by 5% in throughput. In six non-standard scenarios, it outperforms existing systems, with maximum speedups of 6.27× for multimodal inference and 21.4% performance gains on MacBook, demonstrating robustness across model architectures, workloads, and hardware.
- In code editing tasks, VibeServe's speculative decoding reaches 2.9× speedup over baseline, and in prompt caching, it achieves 3.45× throughput. For streaming speech recognition, TTFT improves by 1.69×, and for local image generation, performance increases over fourfold, validating its adaptability and efficiency.
- These results confirm that generation-time system customization can outperform traditional hand-tuned solutions, especially in niche or emerging scenarios, highlighting a new paradigm in AI infrastructure design.
Significance
This work challenges the long-standing reliance on universal runtime stacks by demonstrating that automated, generation-time tailored systems can deliver superior performance and flexibility. It addresses critical bottlenecks in deploying diverse models and workloads, enabling rapid adaptation to new architectures and hardware. The approach reduces engineering effort, accelerates deployment cycles, and paves the way for more intelligent, autonomous AI infrastructure development, with broad implications for industry and research.
Technical Contribution
The paper introduces a novel multi-agent system that orchestrates the end-to-end design of optimized LLM serving stacks. Key innovations include the hierarchical planning and implementation architecture, the integration of a structured communication protocol (MCP), and a comprehensive skills library for rapid adaptation. The system balances correctness verification with performance optimization, enabling scalable generation of bespoke systems across complex model-hardware-workload spaces. Experimental results validate its effectiveness, surpassing traditional methods in multiple challenging scenarios.
Novelty
This is the first work to leverage multi-agent long-horizon planning for end-to-end generation of complete LLM serving systems. Unlike prior approaches focused on kernel or component-level optimization, VibeServe automates the entire system design process, emphasizing customization at generation time rather than runtime. Its hierarchical, collaborative framework represents a significant advancement in AI infrastructure automation, setting a new standard for system-level auto-generation.
Limitations
- Dependence on high-quality reference implementations and correctness checkers may limit applicability in highly complex or novel tasks where validation is difficult.
- Search efficiency in long-horizon planning remains a challenge; large design spaces can slow convergence, requiring further optimization.
- Generated systems can be complex, potentially increasing deployment and maintenance costs, especially in production environments.
Future Work
Future research will focus on enhancing autonomous learning capabilities of the system, integrating reinforcement learning for more efficient search, and expanding the skills library to cover emerging models and hardware. Additionally, efforts will target improving robustness, security, and ease of deployment, aiming to make automated system generation a standard practice in AI infrastructure development.
AI Executive Summary
The rapid evolution of large language models (LLMs) has revolutionized AI applications, yet deploying these models efficiently remains a significant challenge. Traditional systems rely on hand-crafted, general-purpose stacks that are labor-intensive to optimize and often fail to meet the demands of diverse architectures and hardware platforms. As new model types—such as multimodal and hybrid architectures—emerge, the limitations of static, universal runtimes become increasingly apparent.
This paper introduces VibeServe, a pioneering framework that employs multi-agent long-horizon planning to automatically generate bespoke LLM serving systems tailored to specific deployment scenarios. The core architecture comprises an outer planning loop that leverages historical optimization data to select promising design targets, and an inner implementation loop where multiple specialized agents collaboratively produce, validate, and measure candidate systems. Communication among agents is orchestrated via a structured Model Context Protocol (MCP), while a comprehensive skills library provides targeted knowledge on models, hardware, and optimization techniques.
Through iterative refinement, VibeServe can produce highly optimized, end-to-end serving stacks that meet correctness and performance criteria. Experimental evaluations across six diverse scenarios—including standard deployment, code editing, prompt caching, streaming speech recognition, and local image generation—demonstrate its effectiveness. In standard settings, performance matches or exceeds state-of-the-art solutions like vLLM, while in niche scenarios, it achieves up to 6.27× speedups and 21.4% performance improvements.
These results highlight a paradigm shift in AI infrastructure design: moving from runtime generality to generation-time specialization. By automating the end-to-end system creation process, VibeServe reduces engineering effort, accelerates deployment, and enhances adaptability to emerging models and hardware. This approach opens new avenues for scalable, flexible, and intelligent AI service infrastructure, with broad implications for both academia and industry. Future work aims to improve search efficiency, expand the skills library, and increase system robustness, pushing toward fully autonomous AI system generation.
Deep Dive
Plain Language Accessible to non-experts
想象你在一家大型厨房里，厨师们要做出各种不同的菜肴。每次做菜都需要根据菜的类型、用的厨具和食材的不同，调整烹饪方法。以前，厨师们会花很多时间手工设计每道菜的做法，试错不断，效率很低。现在，有个聪明的助手可以根据菜单、厨房设备和食材，自动帮厨师设计出最合适的做法。这个助手会不断试验、验证，确保菜既好吃又快做好。这样，厨房的效率大大提高，菜的质量也更稳定。VibeServe就像这个智能助手，它能自动为不同的模型、硬件和任务定制专属的“菜谱”，让AI服务变得更快、更智能、更个性化。
ELI14 Explained like you're 14
你知道现在很多游戏和聊天机器人用的超级聪明的程序吗？这些程序背后其实有很多复杂的代码在支撑。以前，工程师们要花很长时间手工调试每个程序，确保它们跑得快、答得准，但每次换个场景都要重新调试，特别麻烦。现在，有一种新方法，就像你用手机自动帮你设计最适合你游戏的配置一样，科学家们用一种叫VibeServe的智能系统，让电脑自己设计出最适合不同硬件和任务的“程序”。它像一个超级聪明的助手，可以不断试验、验证，找到最棒的方案。这样一来，不管是在强大的GPU上，还是在普通的笔记本电脑上，都能让AI变得更快、更准、更省力。是不是很酷？未来，我们的AI助手会变得越来越聪明，帮我们解决各种复杂的问题，就像你有个超级帮手一样！
Abstract
For years, we have built LLM serving systems like any other critical infrastructure: a single general-purpose stack, hand-tuned over many engineer-years, meant to support every model and workload. In this paper, we take the opposite bet: a multi-agent loop that automatically synthesizes bespoke serving systems for different usage scenarios. We propose VibeServe, the first agentic loop that generates entire LLM serving stacks end-to-end. VibeServe uses an outer loop to plan and track the search over system designs, and an inner loop to implement candidates, check correctness, and measure performance on the target benchmark. In the standard deployment setting, where existing stacks are highly optimized, VibeServe remains competitive with vLLM, showing that generation-time specialization need not come at the cost of performance. More interestingly, in non-standard scenarios, VibeServe outperforms existing systems by exploiting opportunities that generic systems miss in six scenarios involving non-standard model architectures, workload knowledge, and hardware-specific optimizations. Together, these results suggest a different point in the design space for infrastructure software: generation-time specialization rather than runtime generality. Code is available at https://github.com/uw-syfi/vibe-serve.