Pie: A Programmable Serving System for Emerging LLM Applications
Pie system enables flexible, efficient LLM serving via APIs and inferlets, boosting throughput by 1.3x-3.4x.
Key Findings
Methodology
Pie system decomposes the traditional generation loop into fine-grained service handlers, controlled by user-provided programs called inferlets via an API. Inferlets are executed using WebAssembly, offering a lightweight sandbox environment.
Key Results
- Pie matches state-of-the-art performance on standard tasks with 3-12% latency overhead. In agentic workflows, it improves latency and throughput by 1.3x-3.4x through application-specific optimizations.
- Experimental results show significant performance gains in Graph-of-Thought and agentic workflows.
- Pie achieves notable optimization effects in multi-step generation and custom KV cache strategies.
Significance
Pie addresses the limitations of existing LLM serving systems in handling complex reasoning strategies and agentic workflows by providing flexible generation control and efficient computation. It offers new solutions for academia and industry, especially in applications requiring custom generation logic.
Technical Contribution
By introducing inferlets and a WebAssembly execution environment, Pie offers a programmable model distinct from existing systems, allowing users to implement custom logic and optimization strategies without modifying the serving system.
Novelty
Pie is the first system to fully delegate the LLM generation process to user programs, breaking the constraints of traditional monolithic generation loops and offering unprecedented flexibility and control.
Limitations
- In some complex reasoning scenarios, Pie may require higher programming complexity to achieve optimization.
- The WebAssembly execution environment might limit performance in certain cases.
Future Work
Future work will focus on extending Pie's API to support more LLM architectures and optimizing WebAssembly execution efficiency. Additionally, exploring performance across more application scenarios is planned.
AI Executive Summary
As large language model (LLM) applications diversify, existing serving systems face challenges in handling complex reasoning strategies and agentic workflows. The Pie system enhances flexibility and efficiency by decomposing the traditional generation loop into fine-grained service handlers and controlling the process through user-provided programs called inferlets via an API.
The core technologies of Pie include executing inferlets using WebAssembly, providing a lightweight sandbox environment that allows applications to implement new KV cache strategies and custom generation logic. Experimental results show that Pie matches state-of-the-art performance on standard tasks with 3-12% latency overhead, while in agentic workflows, it improves latency and throughput by 1.3x-3.4x through application-specific optimizations.
The introduction of Pie provides new solutions for academia and industry, particularly in applications requiring custom generation logic. Future work will focus on extending Pie's API to support more LLM architectures and optimizing WebAssembly execution efficiency.
Deep Analysis
Background
With the widespread application of large language models (LLMs), existing serving systems face challenges in handling complex reasoning strategies and agentic workflows. Traditional generation loops are often monolithic, making it difficult to adapt to diverse application needs. In recent years, researchers have proposed various optimization strategies, such as KV cache management and custom generation logic, but these strategies are difficult to implement in existing systems.
Core Problem
Existing LLM serving systems are inefficient and lack flexibility in handling complex reasoning strategies and agentic workflows. The traditional monolithic generation loop limits the ability to customize applications, making it difficult to support diverse application needs.
Innovation
The Pie system introduces inferlets and a WebAssembly execution environment to make the generation process programmable. • Decomposes the traditional generation loop into fine-grained service handlers. • Exposes these handlers to user-provided programs called inferlets via an API. • Executes inferlets using WebAssembly, providing a lightweight sandbox environment.
Methodology
- �� Decomposes the traditional generation loop into fine-grained service handlers. • Exposes these handlers to user-provided programs called inferlets via an API. • Executes inferlets using WebAssembly, providing a lightweight sandbox environment. • Allows applications to implement new KV cache strategies and custom generation logic.
Experiments
The experimental design includes performance testing in standard tasks and agentic workflows. Benchmarks used include Graph-of-Thought and multi-step generation tasks. By comparing with existing systems, Pie's significant improvements in latency and throughput are verified.
Results
Experimental results show that Pie matches state-of-the-art performance on standard tasks with 3-12% latency overhead. In agentic workflows, it improves latency and throughput by 1.3x-3.4x through application-specific optimizations.
Applications
Pie system is suitable for application scenarios requiring custom generation logic, such as complex reasoning strategies and agentic workflows. By providing flexible generation control, Pie can significantly enhance the performance of these applications.
Limitations & Outlook
Pie may require higher programming complexity to achieve optimization in some complex reasoning scenarios. Additionally, the WebAssembly execution environment might limit performance in certain cases.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. Traditional LLM serving systems are like a fixed recipe where you have to follow steps exactly. The Pie system is like an open kitchen where you can adjust ingredients and steps according to your taste. Through the API, you can choose different ingredients (data) and cooking methods (generation logic), and even add new spices (computation and I/O operations) during cooking. This flexibility allows you to create dishes that better suit your taste based on different needs.
ELI14 Explained like you're 14
Imagine you're playing a super complex game. Traditional LLM serving systems are like a level you can only play in a fixed order, while the Pie system is like an open-world game where you can choose tasks and paths freely. Through the API, you can customize your game strategy, choose different gear and skills, and even add new challenges during the game. This flexibility allows you to create a unique gaming experience based on your preferences.
Glossary
LLM Serving System
Backend systems that support large language model applications, responsible for handling generation and inference tasks.
In this paper, LLM serving systems are the application background for Pie.
Inferlets
User-provided programs that control various stages of the generation process.
Pie uses inferlets to make the generation process programmable.
KV Cache
A cache structure used to store intermediate results during the generation process.
Pie allows users to customize KV cache strategies.
WebAssembly
A lightweight sandbox execution environment used to execute inferlets.
Pie uses WebAssembly to execute inferlets.
Agentic Workflows
Complex task flows that require interaction with external systems.
Pie shows significant performance improvements in agentic workflows.
Open Questions Unanswered questions from this research
- 1 How can Pie's performance be further optimized without increasing programming complexity?
- 2 What are the performance bottlenecks of WebAssembly when executing inferlets?
- 3 How can Pie be extended to support more LLM architectures?
Applications
Immediate Applications
Custom Generation Logic
Pie allows developers to customize generation logic according to application needs, enhancing application performance.
Complex Reasoning Strategies
Through Pie, developers can implement complex reasoning strategies such as multi-step generation and Graph-of-Thought.
Long-term Vision
Widespread Industry Applications
The flexibility and efficiency of Pie give it broad application potential across multiple industries.
Abstract
Emerging large language model (LLM) applications involve diverse reasoning strategies and agentic workflows, straining the capabilities of existing serving systems built on a monolithic token generation loop. This paper introduces Pie, a programmable LLM serving system designed for flexibility and efficiency. Pie decomposes the traditional generation loop into fine-grained service handlers exposed via an API and delegates control of the generation process to user-provided programs, called inferlets. This enables applications to implement new KV cache strategies, bespoke generation logic, and seamlessly integrate computation and I/O-entirely within the application, without requiring modifications to the serving system. Pie executes inferlets using WebAssembly, benefiting from its lightweight sandboxing. Our evaluation shows Pie matches state-of-the-art performance on standard tasks (3-12% latency overhead) while significantly improving latency and throughput (1.3x-3.4x higher) on agentic workflows by enabling application-specific optimizations.