PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU
PowerInfer leverages neuron activation power-law distribution, enabling single-GPU inference with up to 11.69× speedup, maintaining accuracy on large LLMs.
Key Findings
Methodology
PowerInfer exploits the power-law distribution of neuron activations, preloading frequently activated 'hot neurons' onto GPU, while less active 'cold neurons' are computed on CPU. It employs adaptive online predictors to forecast neuron activation, reducing data transfer and memory load. The system integrates neuron-aware sparse operators for efficient matrix-vector multiplication at the neuron level. Offline strategies generate neuron placement policies based on activation statistics, formulated as an integer linear programming problem, optimizing hardware resource utilization. During inference, the GPU handles hot neurons, and the CPU manages cold neurons, minimizing PCIe transfers and accelerating large model inference on consumer-grade GPUs.
Key Results
- On a single NVIDIA RTX4090, PowerInfer achieves 13.20 tokens/sec for quantized OPT-175B and 8.32 tokens/sec for non-quantized, outperforming llama.cpp by 11.69×, while maintaining model accuracy. For OPT-30B, performance reaches 82% of A100 levels, significantly reducing inference latency.
- Activation analysis shows 26% of neurons account for 80% of activations across models like OPT-30B and LLaMA2, validating the high sparsity and locality. The online predictor achieves over 93% accuracy in dynamic activation prediction, enabling efficient neuron skipping.
- The hybrid GPU-CPU approach, combined with neuron-level sparse operators, overcomes GPU memory limitations and reduces data transfer bottlenecks, enabling high-speed inference of large models on consumer hardware.
Significance
This work addresses the critical bottleneck of GPU memory and bandwidth in deploying large language models on consumer hardware. By leveraging activation sparsity and locality, PowerInfer enables fast, accurate inference on a single GPU, democratizing access to large models. It bridges the gap between high-end server performance and personal computing, fostering wider adoption of AI technologies in edge devices, private environments, and resource-constrained settings. The approach paves the way for scalable, cost-effective AI deployment, reducing reliance on expensive data center infrastructure.
Technical Contribution
The paper introduces a novel neuron-aware hybrid inference framework based on the power-law distribution of neuron activation. It develops adaptive online predictors to forecast neuron activity, employs neuron-level sparse operators for efficient computation, and formulates offline neuron placement policies via integer linear programming. These innovations collectively enable dynamic, memory-efficient large model inference on consumer-grade GPUs, maintaining high accuracy and speed. The system's modular design allows compatibility across multiple LLM architectures, setting a new standard for edge AI deployment.
Novelty
This is the first comprehensive system to exploit the power-law activation distribution for large model inference on single consumer GPUs. Unlike prior methods relying solely on model compression or layer-wise offloading, PowerInfer dynamically predicts neuron activation and strategically places neurons between GPU and CPU, achieving significant speedups without sacrificing accuracy. Its neuron-aware sparse operators and offline placement optimization distinguish it from existing sparse matrix techniques, offering a practical solution for real-world deployment.
Limitations
- The system heavily depends on the activation sparsity characteristic, which varies across models and tasks. Models with low sparsity or different activation functions may see reduced acceleration.
- The complexity of offline neuron placement and predictor training increases deployment difficulty, especially for custom or novel architectures.
- Performance gains diminish when activation sparsity drops below certain thresholds or in models with dense activation patterns, requiring further adaptation.
Future Work
Future directions include developing more robust and generalizable activation prediction models, extending support to diverse activation functions, and automating offline placement strategies. Integrating multi-GPU coordination and dynamic scheduling could further enhance scalability. Additionally, exploring hardware-aware optimizations and applying the approach to other AI tasks beyond language modeling will broaden its impact.
AI Executive Summary
PowerInfer represents a significant advancement in large language model inference, specifically targeting the constraints of consumer-grade GPUs. By harnessing the inherent sparsity and locality in neuron activations, it introduces a hybrid GPU-CPU architecture that preloads frequently activated neurons onto the GPU, while relegating less active neurons to the CPU. This approach effectively reduces GPU memory demands and minimizes data transfer overhead, enabling high-speed inference of models like OPT-175B on a single NVIDIA RTX4090.
The system employs adaptive online predictors to dynamically forecast neuron activation, combined with neuron-aware sparse operators that perform matrix-vector multiplications at the neuron level. Offline strategies generate optimal neuron placement policies based on activation statistics, formulated as an integer linear programming problem. These innovations collectively allow PowerInfer to achieve up to 11.69× speedup over existing solutions like llama.cpp, while maintaining model accuracy.
Experimental results demonstrate that on a single consumer GPU, PowerInfer delivers 13.20 tokens/sec for quantized OPT-175B and 8.32 tokens/sec for non-quantized models, approaching the performance of high-end data center GPUs. Activation analysis confirms that approximately 26% of neurons are responsible for 80% of activations, validating the sparsity-based approach. The system's compatibility with multiple models and activation functions underscores its versatility.
This work addresses the critical bottleneck of large model deployment, democratizing access to powerful AI models by reducing hardware costs and energy consumption. It opens new avenues for edge AI, private deployment, and resource-efficient inference, promising broader adoption and innovation in AI technology.
Deep Analysis
Background
近年来，大规模语言模型（LLMs）如GPT、LLaMA、OPT等在自然语言处理领域取得突破，推动模型规模不断扩大。高性能训练和推理依赖昂贵的GPU集群，限制了模型的普及。模型压缩、剪枝和量化技术虽有所助益，但在单卡消费级GPU上仍难以高效运行。模型切分和稀疏加速方案虽能缓解部分问题，但受限于GPU内存和带宽，推理速度仍难满足边缘应用需求。PowerInfer基于激活稀疏性和局部性特征，提出新颖的GPU-CPU混合架构，旨在突破这一瓶颈。
Core Problem
核心问题在于大模型参数规模超出单GPU内存，导致数据传输和存储成为瓶颈。传统模型切分和稀疏加速在实际应用中效果有限，尤其在低延迟场景下表现不佳。如何利用激活的稀疏性和局部性特征，实现高效、低成本推理，成为亟待解决的问题。这涉及激活预测、神经元存储、稀疏算子设计及硬件调度等多方面技术难题。
Innovation
创新点包括：1）基于幂律激活分布的神经元预加载策略，有效利用激活局部性；2）引入自适应预测器，动态预测激活神经元，减少GPU内存占用；3）设计神经元感知稀疏算子，提升稀疏矩阵乘法效率；4）结合离线神经元放置策略和整数线性规划，优化GPU-CPU神经元分布。这些创新突破了传统模型切分和稀疏加速的局限，实现单GPU高效推理。
Methodology
- �� 离线阶段：分析激活分布，统计热神经元，预加载到GPU，冷神经元由CPU存储。• 训练自适应预测器，实时预测激活神经元，减少GPU内存占用。• 设计神经元感知稀疏算子，直接在神经元级别进行稀疏矩阵乘法。• 利用离线策略和整数线性规划，生成神经元放置方案，考虑激活频率和硬件带宽。• 运行时：结合预测器和稀疏算子，GPU处理热神经元，CPU处理冷神经元，减少数据传输，提升速度。
Experiments
采用Wikipedia数据集，评估OPT-175B、OPT-30B、LLaMA2等模型在单卡RTX4090上的推理性能。比较基线llama.cpp、FlexGen、DejaVu等方案，测量tokens/sec和延迟。进行激活稀疏性、预测准确率和神经元放置策略的消融实验，验证系统的鲁棒性和适应性。
Results
PowerInfer在RTX4090上对OPT-175B模型实现13.20 tokens/sec（量化）和8.32 tokens/sec（非量化），比llama.cpp快11.69倍。对OPT-30B模型，性能接近A100，达82%的生成速率。激活分析显示26%的神经元贡献80%的激活，验证稀疏性和局部性。系统在保持模型精度的同时，大幅提升推理速度，验证了激活幂律分布的实用性。
Applications
适用于边缘设备、个人电脑和私有云环境，支持多模型、多任务推理。可广泛应用于智能助手、内容生成、边缘AI等场景，降低硬件成本，提升响应速度。未来还可结合多GPU和动态调度，进一步扩展规模和性能。
Limitations & Outlook
依赖激活稀疏性，模型激活特性不同或激活率较低时效果减弱。系统设计复杂，需离线预处理和神经元放置策略，部署难度较大。对于非ReLU激活模型，性能提升有限，需持续优化稀疏预测和算子实现。
Plain Language Accessible to non-experts
想象你在一个工厂里工作，工厂里有很多工人（神经元），每天都在做不同的任务。有些工人（热工人）每天都很忙，几乎一直在工作，工厂提前安排好他们的任务，随时准备好。其他工人（冷工人）只在特定任务时才会工作，平时不用他们。工厂通过提前安排热工人的工作区域，让他们一直待在工厂里，减少等待时间。而冷工人只在需要时才调动到工厂里工作，节省空间和时间。这个方法让工厂运转得更快、更省力。PowerInfer就是用类似的策略，把常用的神经元提前加载到GPU里，不常用的由CPU处理，整体提升推理速度，就像工厂效率大大提高一样。
ELI14 Explained like you're 14
你知道在学校里，有些学生每天都很活跃，总是参与各种活动，而有些学生只在特定的课题上表现出色。PowerInfer就像是老师提前安排那些经常活跃的学生（热神经元）在教室里，随时可以用他们完成任务。而那些偶尔活跃的学生（冷神经元），老师只在需要时才叫他们来帮忙。这样，老师不用每次都叫所有学生来，节省了时间和精力。这种方法让学校的工作变得更快更有效率。PowerInfer用类似的办法，把经常用的神经元提前加载到GPU里，不常用的由CPU处理，整体让大模型的推理变得更快，就像学校效率提升了一样。
Abstract
This paper introduces PowerInfer, a high-speed Large Language Model (LLM) inference engine on a personal computer (PC) equipped with a single consumer-grade GPU. The key principle underlying the design of PowerInfer is exploiting the high locality inherent in LLM inference, characterized by a power-law distribution in neuron activation. This distribution indicates that a small subset of neurons, termed hot neurons, are consistently activated across inputs, while the majority, cold neurons, vary based on specific inputs. PowerInfer exploits such an insight to design a GPU-CPU hybrid inference engine: hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons are computed on the CPU, thus significantly reducing GPU memory demands and CPU-GPU data transfers. PowerInfer further integrates adaptive predictors and neuron-aware sparse operators, optimizing the efficiency of neuron activation and computational sparsity. The evaluation shows that PowerInfer significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy across various LLMs (including OPT-175B) on a single NVIDIA RTX 4090 GPU. For the OPT-30B model, PowerInfer achieves performance comparable to that of a high-end server-grade A100 GPU, reaching 82% of its token generation rate on a single consumer-grade RTX 4090 GPU.