A Simple and Effective Pruning Approach for Large Language Models
Wanda uses input activation and weight magnitude product for training-free pruning, outperforming magnitude pruning significantly.
Key Findings
Methodology
Wanda employs a novel importance metric combining input activation norms and weight magnitudes, comparing weights locally per output neuron. It requires only a single forward pass to estimate activation statistics, drastically reducing computational cost. The importance score for each weight is computed as |Wij|·∥Xj∥2, where ∥Xj∥2 is the input activation norm. We then prune the lowest scoring weights within each output group based on a preset sparsity ratio. This approach leverages the presence of large magnitude features in LLMs, addressing the poor performance of traditional magnitude pruning in these models.
Key Results
- On LLaMA and LLaMA-2, Wanda achieves 50% unstructured sparsity with accuracy close to dense models; for example, LLaMA-7B's zero-shot accuracy drops from 66.97% to near 66%, outperforming magnitude pruning by over 20%. Perplexity on WikiText improves from 17.29 to 7.26. Compared to SparseGPT, Wanda maintains similar or better performance with much lower computational cost.
- Results demonstrate that high sparsity levels (up to 80%) can be achieved without significant performance loss, confirming the existence of effective sparse sub-networks. Structured sparsity yields about 1.6× inference speedup, making it practical for deployment.
- Across different model sizes and sparsity structures, Wanda shows robustness and adaptability. Fine-tuning (LoRA or full) further closes the performance gap, indicating the potential for scalable sparse models.
Significance
This work addresses the longstanding challenge of efficiently pruning large language models without retraining. By exploiting the unique properties of emergent large features, Wanda enables rapid, high-quality sparsification, facilitating deployment in resource-constrained environments. Its simplicity and speed make it suitable for real-time model compression, accelerating the adoption of large models in industry. The approach also offers insights into the internal structure of LLMs, revealing the significance of large magnitude features and their role in model sparsity. Future work could extend this method to more complex architectures and hardware-aware optimizations, further broadening its impact.
Technical Contribution
Wanda introduces a new importance metric based on the product of weight magnitude and input activation norm, enabling efficient local comparison within each output neuron. This design reduces complexity from second-order inverse calculations to simple dot products, allowing pruning in a single forward pass. The method leverages the emergent large features in LLMs, providing a theoretically grounded and practically effective approach to high sparsity. It bridges the gap between simple magnitude pruning and more complex second-order methods, offering a scalable, training-free solution with broad applicability.
Novelty
The key innovation lies in combining input activation norms with weight magnitudes for importance scoring, and performing local per-output pruning rather than global or layer-wise. This exploits the unique property of large magnitude features in LLMs, which traditional magnitude pruning overlooks. Unlike SparseGPT's layer-wise reconstruction requiring iterative weight updates, Wanda's single-pass, no-update approach is both simpler and faster, yet achieves comparable or superior results. This marks a new direction in efficient, training-free large model pruning.
Limitations
- While Wanda performs well at moderate sparsity levels, performance degradation becomes noticeable at very high sparsity (above 80%). Its effectiveness depends on the representativeness of calibration data for activation statistics. The local output grouping strategy may not generalize to all architectures or tasks, potentially limiting its universality. Hardware acceleration for structured sparsity still requires further development to fully realize speedups in practice.
- Additionally, the method assumes the presence of large magnitude features, which may not be universal across all models or domains. Future work should explore adaptive importance metrics and robustness under different data distributions.
Future Work
Future research can focus on integrating Wanda with task-specific fine-tuning methods like LoRA to further boost performance. Developing hardware-aware sparse formats and acceleration techniques will enhance practical deployment. Extending the approach to other model architectures, such as vision transformers, and exploring dynamic sparsity strategies could broaden its applicability. Moreover, theoretical analysis of the emergent features and their role in sparsity can deepen understanding of model internal mechanisms.
AI Executive Summary
Large language models (LLMs) have revolutionized natural language processing but come with prohibitive computational costs due to their massive parameter counts. Traditional compression techniques like quantization and pruning face challenges in scaling to models with billions of parameters. Existing pruning methods such as magnitude pruning and SparseGPT either require extensive retraining or involve complex second-order computations, limiting their practicality for billion-scale models.
This paper introduces Wanda, a novel, training-free pruning approach that leverages the unique properties of LLMs, specifically the emergence of large magnitude features. Wanda computes a simple importance metric by multiplying each weight's magnitude with the input activation norm, estimated via a single forward pass. We compare weights locally within each output neuron, pruning the least important weights according to a preset sparsity ratio. This method is computationally efficient, requiring no retraining or weight updates.
Experimental results on LLaMA and LLaMA-2 demonstrate Wanda's effectiveness. It achieves 50% unstructured sparsity with minimal accuracy loss and significantly better perplexity than magnitude pruning. Compared to SparseGPT, Wanda offers similar performance at a fraction of the computational cost. The method also scales well to structured sparsity, providing up to 1.6× inference speedup.
Wanda's simplicity and efficiency make it highly suitable for real-world deployment, especially in resource-constrained environments. It opens new avenues for understanding and exploiting the internal structure of LLMs, highlighting the importance of emergent large features. Future work will explore combining Wanda with fine-tuning, hardware acceleration, and extending to other architectures, aiming to make large models more accessible and practical.
Deep Analysis
Background
近年来，随着Transformer架构的普及，大规模预训练语言模型（如GPT、BERT、LLaMA）在多个任务中表现出色，但其参数规模不断扩大，带来了计算、存储和能耗的巨大挑战。传统的模型压缩技术包括量化、剪枝和稀疏化，已在小模型中取得一定成功，但在亿级参数的LLMs中，效果受限。剪枝方法如Magnitude、Optimal Brain Damage（OBD）、SparseGPT等，虽然在小模型中表现优异，但在大模型中面临效率瓶颈，尤其是需要复杂的二阶信息或再训练。近年来，研究发现LLMs中出现了大量高幅值特征，揭示了模型内部的稀疏性潜力，为无训练剪枝提供了新可能。
Core Problem
大规模预训练模型的剪枝面临两个核心难题：一是传统幅值剪枝在LLMs中效果不佳，导致性能严重下降；二是高效剪枝方法需在不再训练的前提下，快速找到稀疏子网络。现有方法如SparseGPT虽然不需再训练，但计算复杂度高，难以在实际应用中快速部署。此外，模型内部的特殊特性（如大幅值激活）未被充分利用，限制了剪枝效果。解决这些问题，既要保证剪枝效率，又要保持模型性能，是当前研究的重点。
Innovation
本研究提出的Wanda方法具有三大创新：• 利用输入激活范数与权重幅值的乘积作为重要性指标，结合模型内部特征分布，提升剪枝的准确性；• 在每个输出单元内局部比较权重，避免全局剪枝带来的性能损失；• 仅用单次前向传播估算激活统计，极大降低计算成本。相较于SparseGPT的二阶重构和OBD的全局优化，Wanda更简洁高效，适合大规模模型的快速稀疏化，且无需再训练或二阶逆矩阵计算。
Methodology
- �� 计算每层输入激活的范数（∥Xj∥2）作为激活重要性指标；• 对每个线性层的每个输出单元（行）内，计算每个权重的乘积评分：|Wij|·∥Xj∥2；• 按照预设稀疏比例，将低评分的权重剪除；• 只需一次前向传播获取激活统计，无需再训练或微调；• 适应结构化N:M稀疏，通过每M个连续权重内的评分进行筛选。
Experiments
在LLaMA和LLaMA-2模型上，采用128序列校准数据，评估不同稀疏比例（50%、80%）的性能。对比幅值剪枝和SparseGPT，指标包括准确率和perplexity。通过零样本任务和语言建模验证，Wanda在保持模型性能的同时实现高稀疏率，表现优异。结构化稀疏在推理速度提升方面也显示出明显优势。
Results
Wanda在50%非结构化稀疏下，LLaMA-7B模型的准确率从66.97%降至接近66%，在perplexity上从17.29降至7.26，优于幅值剪枝的17.29。在结构化稀疏中，性能仍优于传统方法。推理加速方面，结构化2:4稀疏实现了1.6倍推理速度提升。微调（LoRA或全参数微调）后，性能差距进一步缩小，验证了稀疏子网络的存在和可调性。
Applications
Wanda适用于模型部署中的快速剪枝，尤其在边缘设备和实时场景中。无需再训练，节省时间和资源。结合微调策略，还能恢复或提升模型性能，适应多任务需求。未来结合硬件优化，有望实现更高效的推理速度和能耗降低。
Limitations & Outlook
当前方法在极高稀疏（80%以上）时，性能仍有下降，且依赖校准数据的代表性。局部输出单元剪枝策略可能不适用于所有模型架构，硬件支持不足也限制了结构化稀疏的实际效果。未来需结合模型微调和硬件优化，提升鲁棒性与实用性。
Abstract
As their size increases, Large Languages Models (LLMs) are natural candidates for network pruning methods: approaches that drop a subset of network weights while striving to preserve performance. Existing methods, however, require either retraining, which is rarely affordable for billion-scale LLMs, or solving a weight reconstruction problem reliant on second-order information, which may also be computationally expensive. In this paper, we introduce a novel, straightforward yet effective pruning method, termed Wanda (Pruning by Weights and activations), designed to induce sparsity in pretrained LLMs. Motivated by the recent observation of emergent large magnitude features in LLMs, our approach prunes weights with the smallest magnitudes multiplied by the corresponding input activations, on a per-output basis. Notably, Wanda requires no retraining or weight update, and the pruned LLM can be used as is. We conduct a thorough evaluation of our method Wanda on LLaMA and LLaMA-2 across various language benchmarks. Wanda significantly outperforms the established baseline of magnitude pruning and performs competitively against recent method involving intensive weight update. Code is available at https://github.com/locuslab/wanda.