LLM-Pruner: On the Structural Pruning of Large Language Models
LLM-Pruner: a structural pruning method that compresses large models with only 50K data in 3 hours, maintaining multi-task performance.
Key Findings
Methodology
This paper introduces the LLM-Pruner framework, which employs gradient-based dependency detection to automatically identify coupled structures within LLMs. Using importance estimation—combining first-order gradients and Hessian approximations—the method selects minimal-impact groups for pruning. The process involves automatic grouping of dependent structures, avoiding manual design. Post-pruning, the model is rapidly fine-tuned with LoRA, requiring only 50K samples and 3 hours, effectively restoring performance. Experiments on LLaMA, Vicuna, and ChatGLM show that pruning 20% of parameters retains 94.97% of original accuracy, demonstrating task-agnostic, low-resource, and efficient compression.
Key Results
- On LLaMA-7B, 20% parameter pruning with minimal fine-tuning maintains 94.97% of original performance, with only 3 hours needed, validating high efficiency and task independence.
- In Vicuna-7B, similar pruning achieves 92.03% of baseline accuracy, with performance degradation less than 8%.
- Ablation studies confirm second-order importance estimation outperforms first-order, with pruning ratios up to 60% still preserving acceptable performance.
Significance
This work addresses the critical bottleneck of large model deployment by drastically reducing data and computational requirements for compression. It enables versatile, task-agnostic model shrinking, facilitating deployment on edge devices and reducing inference latency. The approach bridges the gap between high-performance models and resource-constrained environments, impacting both academia and industry by making large models more accessible and practical.
Technical Contribution
The paper pioneers a fully automated, dependency-aware structural pruning framework for LLMs, integrating gradient and Hessian-based importance estimation with dependency detection. It leverages LoRA for rapid post-training, significantly shortening fine-tuning time and resource needs. The method's novelty lies in automatic structure grouping and minimal data reliance, offering a scalable solution for large-scale model compression with theoretical and practical innovations.
Novelty
This is the first work to realize fully automated, dependency-based structural pruning of LLMs, eliminating manual design of pruning units. By combining gradient importance with Hessian approximations, it achieves high pruning ratios with minimal performance loss, setting a new standard for resource-efficient large model compression.
Limitations
- Performance drops sharply beyond 50% pruning, indicating the need for more robust importance metrics or adaptive strategies.
- Hessian approximation stability may decline in extremely sparse regions, affecting pruning quality.
- Fine-tuning with LoRA, while fast, may still introduce biases or limit generalization across diverse tasks.
Future Work
Future directions include refining dependency detection for higher pruning ratios, integrating multi-task fine-tuning strategies, and exploring self-supervised importance estimation. Extending the framework to multilingual models and dynamic pruning during inference also holds promise for broader applicability.
AI Executive Summary
Large language models (LLMs) have revolutionized natural language processing, yet their enormous size poses significant deployment challenges. Existing compression methods like distillation and quantization often compromise the models' multi-task capabilities or require extensive retraining. To address this, the authors propose LLM-Pruner, a novel framework that combines automatic dependency detection with importance estimation to enable task-agnostic, structural pruning. The core innovation lies in automatically identifying coupled structures within the model using gradient and Hessian-based importance metrics, thus ensuring minimal disruption during pruning. This process is fully automated, eliminating manual design, and is complemented by LoRA-based rapid fine-tuning, which restores performance within 3 hours using only 50K samples. Experimental results on LLaMA-7B, Vicuna-7B, and ChatGLM-6B demonstrate that pruning 20% of parameters retains over 94% of the original performance, with some models even outperforming unpruned counterparts in certain tasks. This approach significantly reduces the resource barrier for deploying large models, making them more accessible for real-world applications. The method's task-agnostic nature and low data requirement open new avenues for scalable model compression, especially in resource-limited environments. Looking ahead, further improvements could involve more sophisticated dependency analysis, multi-task fine-tuning, and extending the framework to multilingual and dynamic inference scenarios. Overall, LLM-Pruner marks a substantial step forward in making large language models more practical, efficient, and versatile.
Deep Dive
Abstract
Large language models (LLMs) have shown remarkable capabilities in language understanding and generation. However, such impressive capability typically comes with a substantial model size, which presents significant challenges in both the deployment, inference, and training stages. With LLM being a general-purpose task solver, we explore its compression in a task-agnostic manner, which aims to preserve the multi-task solving and language generation ability of the original LLM. One challenge to achieving this is the enormous size of the training corpus of LLM, which makes both data transfer and model post-training over-burdensome. Thus, we tackle the compression of LLMs within the bound of two constraints: being task-agnostic and minimizing the reliance on the original training dataset. Our method, named LLM-Pruner, adopts structural pruning that selectively removes non-critical coupled structures based on gradient information, maximally preserving the majority of the LLM's functionality. To this end, the performance of pruned models can be efficiently recovered through tuning techniques, LoRA, in merely 3 hours, requiring only 50K data. We validate the LLM-Pruner on three LLMs, including LLaMA, Vicuna, and ChatGLM, and demonstrate that the compressed models still exhibit satisfactory capabilities in zero-shot classification and generation. The code is available at: https://github.com/horseee/LLM-Pruner