PowerSlider: Exploiting Phase Asymmetry for LLM Serving under Demand Response
PowerSlider exploits phase heterogeneity with online KKT solver to optimize GPU frequency and KV cache under dynamic power caps, maintaining high throughput.
Key Findings
Methodology
PowerSlider integrates phase-aware scheduling by disaggregating the LLM inference pipeline into Prefill, Think, and Answer stages, each with distinct frequency and memory control knobs. It models the power-performance trade-offs using convex relaxation and applies a KKT-based online solver to dynamically adjust per-stage GPU frequencies and KV cache allocations in response to real-time power caps. The Flex SLO contract converts user latency slack into an optimization constraint, enabling controlled degradation during demand response events. The system is implemented on SGLang and evaluated with Azure traces, demonstrating significant throughput gains under deep power caps while maintaining latency tail bounds.
Key Results
- On SGLang benchmarks, PowerSlider achieves 78.3% online goodput at a 30% power reduction, outperforming five baselines with 47.6%, a 1.64× improvement. It sustains 92% mean goodput during a CAISO grid emergency simulation with caps dropping to 0.41×, and keeps latency tails within 1.3× of nominal, vastly superior to baseline 2.3–6×. These results validate the effectiveness of phase heterogeneity exploitation and online convex optimization in real-time power management.
- In real-world production traces, PowerSlider maintains near-peak throughput under variable power caps, effectively handling bursty loads and long reasoning chains. During grid emergency scenarios, it sustains high throughput with minimal latency tail increase, demonstrating robustness. Ablation studies confirm that stage-specific frequency and KV cache control are critical for performance gains. The system adapts seamlessly to changing workload and power conditions, outperforming static and heuristic approaches.
- The experimental results highlight that phase-aware scheduling combined with online convex optimization can significantly improve energy efficiency without sacrificing service quality. The ability to operate under deep power caps while maintaining high throughput and controlled latency tail demonstrates a practical pathway for integrating demand response into large-scale AI inference infrastructure. The approach’s scalability and adaptability make it promising for deployment in diverse hardware and workload scenarios.
Significance
This work addresses a critical challenge in deploying large language models at scale under real-time power constraints driven by grid demand response. By leveraging phase heterogeneity and online convex optimization, PowerSlider provides a novel framework for dynamic, fine-grained power management that preserves inference throughput and latency guarantees. It bridges the gap between static energy optimization and real-time power caps, enabling AI services to operate efficiently in increasingly constrained power environments. The methodology paves the way for greener, more resilient AI deployment, aligning with industry goals of sustainability and operational flexibility. Its principles can extend to other compute-intensive workloads, influencing future research in energy-aware scheduling and hardware-software co-design.
Technical Contribution
The paper introduces a phase heterogeneity-aware scheduling framework that models the inference pipeline as a disaggregated set of stages with distinct frequency and memory control knobs. It formulates the joint resource allocation as a convex optimization problem, solved online via a KKT-based iterative solver with sub-10 ms latency. The Flex SLO contract effectively converts user latency slack into a convex constraint, enabling controlled degradation. The system integrates multi-granularity control—GPU frequency, KV cache partitioning, and request routing—within a unified real-time optimizer. These innovations collectively enable high-performance inference under deep, dynamic power caps, surpassing prior static or coarse-grained approaches.
Novelty
This is the first work to explicitly model and exploit phase heterogeneity in large-scale inference pipelines for dynamic power management. Unlike prior static or heuristic-based methods, PowerSlider uses an online convex optimization framework with stage-specific control knobs, guided by a convex relaxation of the resource allocation problem. Its integration of Flex SLO contracts with real-time KKT solvers for power-aware scheduling under demand response environments represents a novel paradigm shift in energy-efficient AI inference. This approach fundamentally advances the state-of-the-art in real-time, fine-grained power management for large models.
Limitations
- The approach relies on hardware support for fine-grained frequency and cache control; in hardware with limited DVFS granularity, performance gains may diminish. Further hardware-software co-design is needed.
- The online KKT solver, while fast, may face scalability challenges with increasing workload complexity or multiple simultaneous constraints, requiring more scalable algorithms.
- Current evaluation focuses on GPU-based inference; extending to other hardware architectures or multi-modal models requires additional adaptation and validation.
Future Work
Future research will explore integrating hardware power gating and dynamic voltage/frequency scaling at finer granularity, further reducing static power overhead. Extending the framework to multi-modal and multi-task workloads, incorporating multi-objective optimization (e.g., energy, latency, cost), and deploying in real-world cloud environments are promising directions. Additionally, developing hardware-aware scheduling algorithms that leverage emerging power management features will enhance robustness and efficiency in diverse deployment scenarios.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
Imagine you’re managing a busy kitchen with many chefs preparing different dishes. Some chefs need to work quickly to finish their tasks, while others can slow down without affecting the overall meal. Sometimes, the electricity supply to the kitchen gets limited—like during a power outage—so you need to decide how to allocate energy to each chef without ruining the meal.
PowerSlider is like a smart manager who knows which chefs can slow down or speed up, and adjusts their work based on how much power is available. For example, the chef preparing ingredients (prefill) needs to work fast, but the chef designing the dish (thinking) can slow down a bit. The chef assembling the final dish (answering) can also operate at a lower speed if needed.
When power is tight, the manager might tell some chefs to pause or work slower, focusing energy on the most important tasks. But the manager does this very carefully, using real-time information about how much energy is left and how much each task can tolerate slowing down. This way, the kitchen keeps running smoothly, producing good meals without wasting energy.
In essence, PowerSlider is like a clever kitchen supervisor who dynamically balances energy and work, ensuring the meal gets prepared efficiently even when power is scarce. It’s about smartly adjusting each part of the process, so everything works well together, saving energy while still delivering tasty results.
Abstract
AI inference clusters are increasingly constrained by instantaneous power, not just energy: grid operators condition new capacity on demand response, imposing time-varying power caps. Existing LLM serving systems optimize a static energy objective or shed fixed priority tiers under load; either way, goodput collapses when the power envelope moves. An LLM pipeline is not a uniform load: compute-bound prefill loses throughput almost linearly with GPU frequency, memory-bound answer decode sustains it down to $0.57\times$ nominal, and reasoning's thinking phase couples KV-cache capacity to scheduling -- so a cap should be steered to where each watt costs the least performance. PowerSlider does so with a new Flex SLO contract that turns bounded user slack into an optimization constraint, prefill--think--answer disaggregation exposing per-stage frequency and KV control, and a Karush--Kuhn--Tucker (KKT) online solver re-solving within 7.7 ms of every cap change, backed by a consolidated fail-safe that power-gates drained instances when DVFS bottoms out on static power. On SGLang with production traces, \sys{} sustains 78.3\% online goodput at a 30\% cap reduction versus 47.6\% for the best of five baselines ($1.64\times$), holds latency-critical tails within $1.3\times$ of nominal (baselines: $2.3$--$6\times$, up to $12\times$), and delivers 92\% mean goodput through a replayed CAISO grid-emergency day bottoming at $0.41\times$ (54\% at the trough; every baseline below 7\%).