Adaptive Multi-Objective Tiered Storage Configuration for KV Cache in LLM Service
Kareto uses simulation-based Pareto optimization to dynamically configure multi-tier storage for KV caches, improving efficiency.
Key Findings
Methodology
This work employs a high-fidelity end-to-end simulator combined with a Pareto optimization framework to explore the storage configuration space across GPU HBM, host DRAM, and disk. Kareto integrates a diminishing-return-guided pruning strategy to efficiently narrow the search, coupled with a fine-grained adaptive tuner that adjusts TTL policies and access patterns based on workload characteristics. The simulation accurately models cloud storage costs and performance, enabling the system to identify Pareto-optimal configurations that balance cost, latency, and throughput. The approach avoids the limitations of analytical models by capturing complex nonlinear interactions and discontinuities, providing a practical solution for real-world deployment.
Key Results
- On real production traces, Kareto automatically adapts configurations, achieving up to 9.3% throughput increase, 58.3% latency reduction, or 20.2% cost savings compared to fixed 1024GB DRAM setups.
- Hybrid configurations, such as 256GB DRAM combined with cloud disks, outperform single-tier setups in cost-performance trade-offs, validated through extensive simulation.
- Fine-grained TTL tuning and simulation-driven search significantly improve cache hit rates and resource utilization, demonstrating robust adaptability across diverse workloads.
Significance
This research addresses a critical bottleneck in deploying large language models at scale, providing an intelligent, workload-aware storage configuration mechanism. It advances the state-of-the-art by integrating high-fidelity simulation with multi-objective Pareto optimization, enabling cloud providers and enterprises to achieve cost-effective, high-performance inference services. The approach overcomes the nonlinear and discontinuous nature of storage performance and cost relationships, offering a practical pathway toward autonomous resource management in complex AI infrastructure.
Technical Contribution
The paper introduces a novel framework combining detailed simulation with Pareto-based multi-objective optimization, incorporating a diminishing-return-guided pruning method to efficiently explore large configuration spaces. It proposes a prefix-tree-based TTL adjustment mechanism for workload-specific cache management, significantly improving cache efficiency and resource utilization. The high-fidelity simulator models real-world cloud storage costs and performance, enabling precise evaluation of diverse configurations, a key step beyond traditional analytical models that struggle with nonlinearities and phase transitions.
Novelty
This is the first work to systematically combine simulation-driven multi-objective Pareto optimization with workload-adaptive cache management for tiered storage in LLM inference. Its core innovations—recursive pruning based on diminishing returns and workload-aware TTL tuning—set it apart from existing static or heuristic approaches, offering a scalable, flexible solution for complex, real-world scenarios.
Limitations
- The simulation-based approach requires significant computational resources, limiting real-time deployment feasibility without further optimization.
- Dependence on historical workload traces may reduce effectiveness during sudden workload shifts or novel request patterns.
- Cloud storage price fluctuations and hardware heterogeneity pose challenges for maintaining up-to-date models and optimal configurations.
Future Work
Future efforts will focus on integrating online learning algorithms for real-time adaptation, extending the framework to multi-cloud and multi-region environments, and reducing simulation costs through surrogate modeling. Additionally, exploring reinforcement learning-based policies for dynamic cache management could further enhance system responsiveness and robustness.
AI Executive Summary
The rapid growth of large language models (LLMs) has revolutionized AI applications, but deploying these models efficiently at scale remains a challenge. Central to this challenge is the management of key-value (KV) caches, which store intermediate representations to accelerate inference. Traditional approaches rely on fixed storage configurations, typically involving GPU high-bandwidth memory (HBM), host DRAM, and disk, but these static setups often lead to resource underutilization or bottlenecks amid dynamic workloads.
This paper introduces Kareto, a novel framework that leverages high-fidelity simulation and multi-objective Pareto optimization to dynamically configure tiered storage for KV caches. By accurately modeling the complex interactions among storage tiers, workload patterns, and cloud pricing, Kareto identifies optimal configurations that balance cost, latency, and throughput. Its core components include a simulation engine that replicates real-world inference behavior, a diminishing-return-guided pruning strategy to efficiently explore the vast configuration space, and a workload-adaptive TTL tuner that fine-tunes cache management policies.
Experimental results on real production traces demonstrate Kareto's ability to adapt to diverse workload characteristics, outperforming static configurations by significant margins. It achieves up to 9.3% higher throughput, reduces latency by over 58%, and cuts operational costs by 20% in various scenarios. These improvements are critical for cloud providers and enterprises aiming to deploy large-scale LLM services cost-effectively while maintaining high performance.
The significance of this work lies in its holistic approach to the multi-objective optimization problem, overcoming the nonlinear and discontinuous relationships inherent in storage performance and costs. By integrating simulation with intelligent search, Kareto paves the way for autonomous, workload-aware resource management in AI infrastructure. Future directions include online learning integration, multi-cloud extension, and real-time adaptive policies, promising even more resilient and efficient AI deployment strategies.
Deep Analysis
Background
近年来，随着大规模语言模型（LLMs）在自然语言处理中的广泛应用，推理服务的效率成为研究焦点。KV缓存技术通过存储前序请求的键值对，显著降低了模型的推理延迟，成为核心优化手段。早期工作如Transformer缓存机制、vLLM、Mooncake等，主要关注缓存替换策略和固定容量配置，但未充分考虑多层存储体系的异质性和动态变化。随着GPU HBM、主存和磁盘的逐步普及，如何智能调度和配置存储资源，成为提升系统性能和降低成本的关键。传统方法多为静态预设，难以应对请求模式的变化，导致资源浪费或性能瓶颈。近年来，学者们开始尝试动态调节存储配置，但缺乏系统性、端到端的优化框架，限制了实际应用的效果。
Core Problem
核心问题在于如何在多层存储体系中，动态、智能地配置存储容量和调度策略，以实现成本、延迟和吞吐的最优平衡。现有方案多为静态配置，不能适应请求的多变性，导致资源利用率低、性能不稳定。存储层的非线性关系和突变行为（如容量饱和、价格跳跃）使得传统线性模型难以准确预测系统表现。如何在复杂的非线性、多目标耦合中找到最优配置，成为亟待解决的难题。
Innovation
本文提出结合高保真模拟和多目标帕累托优化的系统框架，创新性地引入递减收益引导的剪枝策略，有效探索庞大配置空间。设计了基于前缀树的TTL调优机制，根据访问频率动态调整存活时间，提升存储效率。利用模拟器精确反映云存储成本与性能关系，避免了传统模型中难以捕捉的非线性和突变行为。系统实现了存储配置的弹性调度，显著优于静态预设方案，为多目标优化提供了新思路。
Methodology
- �� 构建高保真端到端模拟器，模拟存储层次（HBM、主存、磁盘）中的KV存取行为，结合云存储价格模型。• 设计多目标帕累托优化框架，将成本、延迟、吞吐作为目标，利用模拟结果评估配置优劣。• 引入递减收益引导的剪枝策略，逐步缩小搜索空间，重点探索高敏感区域。• 采用前缀树分析，动态调整KV块的TTL值，根据访问频率和重用特征，优化存储容量分配。• 结合历史轨迹，利用模拟结果自动识别最优配置集，满足不同用户需求。
Experiments
使用真实生产轨迹，包括多种请求类型和请求量，评估Kareto在不同场景下的表现。比较静态配置、启发式调度和Kareto的性能差异。指标涵盖吞吐量、延迟（TTFT）、存储成本和命中率。通过多轮调优，验证算法在不同负载和存储层组合下的适应性。结果显示，Kareto在多场景中均优于基线方案，特别是在高负载环境下，性能提升明显。
Results
在真实轨迹上，Kareto实现最高9.3%的吞吐提升，延迟降低58.3%，成本节省20.2%。混合存储配置（如256GB DRAM+云磁盘）在成本与性能间找到最佳折中。调优机制显著提升缓存命中率，减少存储浪费。模拟验证了系统在不同工作负载中的自适应能力，展现出优越的资源利用效率和弹性调度能力。
Applications
该方案适用于云端大规模LLM推理服务，特别是在多租户环境中实现弹性资源调度。可广泛应用于AI基础设施、云服务提供商、企业内部AI平台，提升存储利用率和服务质量。实现条件包括对工作负载的历史轨迹数据和云存储价格模型的掌握，系统可根据实际需求自动调整配置。
Limitations & Outlook
模型依赖历史轨迹，可能在突发或极端场景表现不足。模拟计算成本较高，实时调度仍需优化。对云价格波动和硬件异质性敏感，需持续更新模型参数。未来将结合在线学习机制，提升系统的实时适应能力，降低计算成本。
Plain Language Accessible to non-experts
想象你在经营一家餐厅，菜单上有多种菜肴（存储层），每种菜的成本和受欢迎程度不同。你希望用最少的钱做出最受顾客喜爱的菜，但每次顾客点菜（请求）都不一样。过去你用固定菜单（静态配置），但每次都不一定合适。有时候菜单太贵，利润低；有时候菜不够多，顾客等太久。现在，你用一台智能厨房（模拟器）模拟不同菜单组合，观察哪些搭配能让利润最大、菜品新鲜、等待时间短。你还让厨房根据顾客的点菜习惯，调整菜品的供应时间（TTL），确保每次都能快速满足顾客需求。通过不断试验和调整，你找到了一份最优菜单方案，既省钱，又快，又受欢迎。这就像Kareto在存储层中不断试验，找到最合适的配置，让云端AI服务更高效、更经济。
ELI14 Explained like you're 14
想象你在学校的食堂里点餐，你可以选择不同的菜（存储层），有的便宜但慢，有的贵但快。以前，食堂老板总是用同样的菜单，不管当天的学生多还是少。有时候菜太多，浪费钱；有时候菜太少，学生等太久。现在，有个聪明的机器人厨师（模拟器），它可以帮你试各种菜单组合，看看哪个既省钱，又能让学生最快吃到饭。它会根据每天的学生点餐习惯，调整菜的供应时间（TTL），确保每个人都能快快吃到，又不浪费食材。经过多次试验，机器人找到最棒的菜单方案，既省钱又快，还能让学生满意。这个过程就像Kareto在云端存储中不断试验配置，找到最合适的存储方案，让AI服务变得更快、更便宜、更智能。
Glossary
Pareto Front（帕累托前沿）
在多目标优化中，表示在所有目标间达到最优折中的一组解，没有任何目标可以改善而不影响其他目标。
用来描述存储配置在成本、延迟和吞吐的平衡点。
KV Cache（键值缓存）
存储Transformer模型中前序请求的键值对，以避免重复计算，提高推理速度。
是LLM推理中的核心技术之一。
多目标帕累托优化（Multi-objective Pareto Optimization）
在多个目标间寻找非支配解集，确保没有方案在所有目标上都优于另一方案。
用于系统配置的多目标调优。
TTL（存活时间）
键值对在存储中的最大存活时间，超时后会被淘汰。
调节存储空间和缓存效率的重要参数。
模拟器（Simulator）
通过仿真模型重现系统行为，用于评估不同配置的性能和成本。
是优化框架的核心工具。
Open Questions Unanswered questions from this research
- 1 如何在极端突发负载下保持优化效果？未来需结合在线学习机制，实时调整配置策略。
- 2 多云环境中的存储调度如何实现跨平台资源的最优配置？这是未来研究的重要方向。
Applications
Immediate Applications
云端LLM推理优化
云服务提供商可利用Kareto实现弹性存储调度，提升资源利用率，降低成本，改善用户体验。
企业AI基础设施管理
企业内部AI平台通过动态配置存储资源，应对不同业务场景的需求变化，提升效率。
Long-term Vision
智能云基础设施
未来实现全自动、多云、多区域的存储资源调度，推动云计算向自适应、弹性方向发展。
Abstract
The memory-for-computation paradigm of KV caching is essential for accelerating large language model (LLM) inference service, but limited GPU high-bandwidth memory (HBM) capacity motivates offloading the KV cache to cheaper external storage tiers. While this expands capacity, it introduces the challenge of dynamically managing heterogeneous storage resources to balance cost, throughput, and latency under varying workloads. We formulate this as a multi-objective optimization problem: identifying the Pareto frontier across these metrics within the storage configuration space. Using a high-fidelity end-to-end simulator, we observe that the objective functions are non-analytic and exhibit complex variable coupling, making the Pareto frontier difficult to approximate analytically. To obtain the frontier, we introduce Kareto, a KV-cache Adaptive REsource managemenT Optimizer. Kareto leverages a diminishing-return-guided pruning method to efficiently navigate the large configuration space and approximate the Pareto frontier. Additionally, it incorporates a fine-grained adaptive tuner that uses eviction policies in tier storage and KV block access patterns for group-specific cache management, improving cache efficiency. Experiments on real-world traces show that Kareto adapts to workload and can identify configurations of better cost efficiency, covering static strategies. Compared to the fixed setup with 1024 GB DRAM, Kareto can improve throughput by up to 9.3%, or reduce latency by up to 58.3%, or lower cost by up to 20.2% under respective optimization objectives.