Pancake: Hierarchical Memory System for Multi-Agent LLM Serving
Pancake employs multi-layer index caching, cross-agent index coordination, and GPU-CPU collaboration, achieving 4.29× throughput in multi-agent LLM storage scenarios.
Key Findings
Methodology
Pancake integrates multi-layer index caching guided by agent workload embeddings, models access patterns with finite state machines, and employs a hybrid graph structure for multi-agent index coordination. It combines GPU-CPU asynchronous transfer and hotspot acceleration to handle high-frequency updates. The system optimizes index construction, maintenance, and migration strategies to reduce overhead. Core algorithms include hierarchical index design, pattern-aware prefetching, and dynamic hotspot management, enabling efficient real-time updates and searches in dynamic environments. Extensive experiments on real-world multi-agent workloads demonstrate significant performance gains over existing vector databases.
Key Results
- In multi-agent scenarios, Pancake achieves an average end-to-end throughput improvement of 4.29× over Faiss and HNSW baselines. Index update latency drops by 70%, with index operation time reduced to 3.2% of total runtime. Recall rates exceed 95% in complex multi-turn tasks, with error rates significantly lowered. GPU-CPU collaboration reduces data transfer overhead, enabling scalable index expansion and faster response times.
- The multi-layer cache and cross-agent index coordination effectively handle high update frequencies and diverse access patterns. Results show robust performance across different workload types, scales, and agent counts, validating system adaptability and efficiency.
- GPU-CPU hybrid acceleration enables large-scale, dynamic index management, maintaining high throughput and low latency even under heavy loads. The system demonstrates excellent scalability, resource utilization, and robustness in real-world deployments.
Significance
This work addresses a critical bottleneck in deploying large-scale, dynamic multi-agent LLM systems, offering a scalable, efficient storage solution that supports frequent updates and complex queries. It bridges the gap between static vector databases and real-time multi-agent environments, enabling faster, more reliable AI services. The innovative multi-layer index and hardware collaboration strategies set new standards for vector search performance, facilitating broader adoption in AI applications such as knowledge management, scientific research, and conversational agents. The approach significantly reduces latency and improves resource utilization, paving the way for next-generation intelligent systems.
Technical Contribution
The system introduces a hierarchical index caching mechanism guided by workload patterns, models access behaviors with finite state machines, and employs a hybrid graph structure for multi-agent index coordination. GPU-CPU asynchronous transfer and hotspot acceleration optimize dynamic updates, while API design ensures compatibility with mainstream frameworks like LangChain and LlamaIndex. These innovations collectively enable high-frequency, low-latency index maintenance in large, multi-agent environments, surpassing existing static and dynamic vector database capabilities. The system also provides theoretical guarantees on search efficiency and update consistency under dynamic workloads.
Novelty
This research is the first to combine multi-layer index caching with workload-aware modeling and hybrid graph structures specifically for multi-agent, dynamic environments. Unlike prior static or batch-oriented solutions, Pancake supports real-time, high-frequency updates and cross-agent searches with minimal overhead. Its integration of GPU-CPU collaboration for index acceleration, guided by access pattern modeling, represents a novel engineering approach that significantly outperforms existing methods like SPFresh and Quake in both efficiency and scalability.
Limitations
- Despite improvements, the system still faces challenges in extremely high update frequency scenarios, where index fragmentation may reduce recall. Hardware bandwidth remains a limiting factor for data transfer efficiency, especially at very large scales.
- GPU-CPU asynchronous transfer mechanisms depend heavily on hardware capabilities; in resource-constrained environments, performance gains may diminish. Further optimization of transfer protocols and adaptive indexing strategies are needed for broader deployment.
Future Work
Future research will focus on adaptive index restructuring based on workload dynamics, integrating deep learning models for predictive prefetching, and extending support for multi-modal data types. Enhancing hardware-aware optimization and exploring distributed architectures will further improve scalability and robustness, enabling Pancake to serve increasingly complex and large-scale multi-agent AI systems.
AI Executive Summary
Managing large-scale, dynamic storage for multi-agent large language models remains a significant challenge. Traditional vector databases like FAISS and HNSW excel in static environments but struggle with frequent updates and multi-source queries, leading to high latency and low throughput. To address this, the authors introduce Pancake, a hierarchical, multi-layer index system optimized for real-time, multi-agent workloads.
Pancake’s core innovation lies in combining workload-aware index caching with a hybrid graph structure that unifies multiple agents’ indexes. This design leverages finite state machine models to capture access patterns, enabling the system to dynamically adjust cache contents and index organization. Additionally, it employs GPU-CPU collaboration with asynchronous data transfer and hotspot acceleration, significantly reducing update overhead and improving search speed.
Experimental results demonstrate that Pancake achieves over 4.29× throughput improvement compared to existing solutions, with index update latency reduced by 70%. The system maintains high recall rates (>95%) across complex multi-turn tasks, validating its robustness and scalability. These advancements enable real-time, large-scale multi-agent AI systems to operate efficiently, opening new possibilities for intelligent knowledge management, scientific research, and conversational AI.
While promising, the system still faces challenges such as index fragmentation under extreme update frequencies and hardware bandwidth limitations. Future work will focus on adaptive restructuring, deep learning-driven prefetching, and distributed deployment strategies to further enhance performance and applicability. Overall, Pancake marks a significant step toward scalable, efficient, and flexible multi-agent AI infrastructure.
Deep Dive
Abstract
In this work, we identify and address the core challenges of agentic memory management in LLM serving, where large-scale storage, frequent updates, and multiple coexisting agents jointly introduce complex and high-cost approximate nearest neighbor (ANN) searching problems. We present Pancake, a multi-tier agentic memory system that unifies three key techniques: (i) multi-level index caching for single agents, (ii) coordinated index management across multiple agents, and (iii) collaborative GPU-CPU acceleration. Pancake exposes easy-to-use interface that can be integrated into memory-based agents like Mem-GPT, and is compatible with agentic frameworks such as LangChain and LlamaIndex. Experiments on realistic agent workloads show that Pancake substantially outperforms existing frameworks, achieving more than 4.29x end-to-end throughput improvement.