Characterizing CPU-Induced Slowdowns in Multi-GPU LLM Inference
This study reveals CPU resource scarcity causes up to 7.11× increase in first-token latency in multi-GPU LLM inference, emphasizing CPU's critical role.
Key Findings
Methodology
Using systematic experiments combined with real cluster scheduling data, the authors evaluate the impact of varying CPU allocations on multi-GPU inference. They measure key latency components such as Tokenization, Kernel Launch, and synchronization delays across models like Llama 3.1 70B and Qwen 3 30B. The study employs vLLM framework, simulating different CPU core counts (4 to 64) under various request loads and context lengths. By analyzing request timeout rates, GPU utilization, and latency metrics, they identify the mechanisms by which CPU resource constraints induce performance degradation, validating the cost-effectiveness of CPU expansion.
Key Results
- Experiments show that increasing CPU cores from 4 to 32 reduces first-token latency (TTFT) by up to 7.11× under high load and long contexts. CPU-starved configurations frequently cause request timeouts, especially in multi-turn, long-context scenarios. Even with CUDA Graphs and process isolation, CPU bottlenecks persist, indicating hardware configuration is a key factor. The results demonstrate that adding CPU cores is a highly cost-effective way to improve responsiveness and throughput, outperforming GPU scaling in many cases.
- Under heavy request loads, CPU contention causes kernel launch delays and synchronization stalls, leading to GPU idle time and throughput drops. The study finds that in multi-turn conversations, tokenization and synchronization tasks dominate CPU time, creating bottlenecks. Increasing CPU resources reduces these delays, maintaining high GPU utilization and system stability. The analysis of real-world cluster data confirms that CPU underprovisioning is common, and expanding CPU cores offers significant performance gains at low additional cost.
- The study highlights that models like MoE are more sensitive to CPU scarcity than dense models, reaching saturation at lower request rates. The experiments show that CPU resource expansion not only reduces latency but also decreases request timeout rates, making it a practical solution for cloud deployment. The findings provide a comprehensive understanding of CPU-GPU interactions, guiding resource scheduling policies for large-scale inference systems.
Significance
This research challenges the GPU-centric view of inference bottlenecks, demonstrating that CPU resources critically influence multi-GPU large model performance. By systematically analyzing the delay mechanisms, it offers new insights into optimizing resource allocation, reducing latency, and improving stability in cloud and enterprise environments. The findings underscore the importance of balanced CPU-GPU provisioning, especially in multi-tenant settings where resource contention is common. Implementing CPU expansion strategies based on this work can lead to more reliable, cost-effective deployment of large models, accelerating their adoption in real-world applications. The study also opens avenues for future hardware-software co-design to further mitigate CPU bottlenecks.
Technical Contribution
The paper introduces a multi-path analysis framework that links CPU resource constraints to specific latency components—Tokenization, Kernel Launch, and synchronization delays—in multi-GPU inference. It combines empirical data from real cluster logs with controlled experiments, quantifying the cost-benefit ratio of CPU core expansion. The authors demonstrate that CPU bottlenecks cause cascading delays, even in optimized environments with CUDA Graphs and process isolation. They propose a detailed mechanism model explaining how limited CPU cores inflate collective synchronization time and tail stalls, providing a theoretical basis for resource-aware scheduling. This work advances understanding of CPU-GPU interplay and offers practical guidelines for resource provisioning in large-scale inference systems.
Novelty
This is the first comprehensive study to systematically quantify and analyze CPU resource scarcity effects on multi-GPU large language model inference. Unlike prior work focusing solely on GPU hardware improvements, this research emphasizes CPU's critical role in request scheduling, tokenization, and synchronization. It combines real-world cluster data with experimental validation, revealing that CPU underprovisioning leads to significant latency spikes and request timeouts, even with advanced GPU optimizations. The innovative framework and cost-benefit analysis establish CPU expansion as a practical, low-cost performance enhancement strategy, providing a new perspective on optimizing large-scale inference deployments.
Limitations
- The experiments are primarily conducted on specific hardware platforms (H100, H200, RTX Pro 6000) and models (Llama, Qwen). Different architectures or models may exhibit different bottleneck characteristics.
- The study assumes static resource allocation, not accounting for dynamic scheduling or multi-tenant interference in real cloud environments, which could influence results.
- While CPU expansion is cost-effective, in multi-node or NUMA systems, memory access latency and interconnect bandwidth may limit scalability. Further hardware-aware optimization is needed.
Future Work
Future research will explore adaptive, dynamic resource scheduling algorithms that respond to workload variations in real time. Integrating hardware innovations such as high-bandwidth memory and heterogeneous cores could further alleviate CPU bottlenecks. Extending analysis to multi-node, multi-tenant environments will help develop robust, scalable deployment strategies. Additionally, investigating model sparsity and alternative architectures (e.g., mixture-of-experts) under CPU constraints can optimize performance further. The goal is to develop intelligent, cost-efficient resource management frameworks that maximize inference throughput while minimizing latency across diverse deployment scenarios.
AI Executive Summary
The rapid growth of large language models (LLMs) has driven the deployment of multi-GPU systems to meet computational demands. Traditionally, research has focused on GPU hardware improvements, such as tensor cores and high-bandwidth interconnects, to boost inference speed. However, recent findings reveal that CPU resources—often underprovisioned—play a crucial role in limiting overall system performance. This study systematically investigates how CPU bottlenecks, especially in tokenization, kernel launches, and synchronization, cause significant delays even when GPUs are underutilized.
Using a combination of real cluster data and controlled experiments with models like Llama 3.1 70B and Qwen 3 30B, the authors demonstrate that increasing CPU cores from 4 to 32 can reduce first-token latency by up to 7.11× under high load and long contexts. These improvements are consistent across different hardware platforms and optimization strategies, including CUDA Graphs. The research highlights that in multi-turn, long-context scenarios, CPU tasks such as tokenization and request scheduling dominate the latency profile, causing GPU idle time and request timeouts.
The implications are profound for cloud providers and enterprise AI services. Proper CPU provisioning—cost-effective compared to GPU scaling—can dramatically enhance inference responsiveness and stability. The findings advocate for resource-aware scheduling policies that balance CPU and GPU resources, especially in multi-tenant environments where resource contention is common. Future directions include integrating hardware innovations and adaptive scheduling algorithms to further mitigate CPU bottlenecks, enabling scalable, efficient deployment of large models in real-world applications.
Deep Dive
Abstract
Large-scale machine learning workloads increasingly rely on multi-GPU systems, yet their performance is often limited by an overlooked component: the CPU. Through a detailed study of modern large language model (LLM) serving workloads, we find that multi-GPU performance often degrades not because GPUs are saturated, but because CPUs fail to keep them busy. Under limited CPU allocations, systems exhibit symptoms such as delayed kernel launch, stalled communication, and increased tokenization latency, leading to severe GPU underutilization even when ample GPU resources are available. The problem becomes more severe in agentic LLM serving, where long accumulated contexts increase CPU-side tokenization work while high prefix-cache reuse across multi-turn interactions reduces GPU-side prefill work. These bottlenecks persist even in serving stacks that employ process-level separation and modern GPU-side optimizations such as CUDA Graphs. Since CPU cores cost orders of magnitude less than GPUs, provisioning additional cores is a highly cost-effective mitigation. Under moderate serving load, we observe that CPU-starved configurations frequently time out, while providing adequate CPU resources restores responsiveness and reduces time-to-first-token (TTFT) latency by 1.47-7.11x across configurations, all without requiring additional GPUs.