Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization
Týr-Pruner employs end-to-end global sparsity distribution optimization via supernet construction and evolutionary search, retaining 97% performance with 50% parameter removal.
Key Findings
Methodology
Týr-Pruner adopts an end-to-end search framework, constructing a supernet by applying local pruning across layers at multiple sparsity ratios. It uses Taylor expansion-based first and second-order information to identify redundant structures and adjust weights. An expectation error accumulation mechanism ensures balanced error propagation across sparse substructures. The approach employs iterative coarse-to-fine sparsity refinement via evolutionary search, where candidate subnets are evaluated based on a distillation-inspired metric that measures similarity to the dense model. This process efficiently converges to an optimal global sparsity distribution, enabling high compression with minimal performance loss.
Key Results
- In Llama-3.1-70B, the method achieves 50% parameter pruning while maintaining 97% of the original performance, with perplexity at 5.84 on WikiText2, outperforming previous SOTA. For 37.5% pruning, perplexity is only 9.16, with downstream accuracy at 64.15%. On 8B models, similar trends show minimal performance degradation at high sparsity levels.
- The approach requires only 4 million tokens for calibration, significantly reducing data and computational costs compared to prior methods. It demonstrates superior stability and effectiveness across multiple large models and tasks.
- Experimental results validate that the combination of Taylor-based redundancy estimation, error accumulation, and iterative search yields better performance and efficiency than traditional two-stage or gradient-based methods, especially under aggressive sparsity ratios.
Significance
This work advances model compression by enabling fully automated, end-to-end global structural pruning for massive language models. It addresses longstanding challenges such as structure dependency neglect, optimization discontinuity, and high computational costs. The framework's ability to retain nearly full performance at high sparsity levels paves the way for deploying large models on resource-constrained devices, democratizing AI access. Its innovative combination of supernet construction, Taylor-based pruning, and evolutionary search sets a new standard for scalable, intelligent model compression, impacting both academia and industry.
Technical Contribution
Týr-Pruner introduces a novel supernet-based framework integrating Taylor expansion for precise redundancy estimation, along with an expectation error accumulation mechanism to balance multi-structure error propagation. Its key technical innovation lies in the iterative coarse-to-fine sparsity search via evolution, significantly reducing search space and improving convergence speed. The method's end-to-end nature and minimal calibration data requirement distinguish it from gradient-dependent or heuristic approaches, offering a robust, scalable solution for large-scale model compression.
Novelty
This is the first comprehensive framework combining Taylor-based local pruning, expectation error balancing, and iterative evolutionary search for global structural pruning of massive language models. Unlike prior methods that rely on separate importance evaluation and two-stage optimization, Týr-Pruner performs end-to-end learning of the optimal sparsity distribution, effectively addressing structure dependencies and optimization discontinuities. Its supernet construction and coarse-to-fine search paradigm represent significant innovations in the field.
Limitations
- While effective at high sparsity levels, the method's performance may still degrade under extreme compression (>75%), indicating limits in redundancy removal and model expressiveness.
- Dependence on calibration data quality could influence the accuracy of redundancy estimation and sparsity optimization, especially with biased or limited datasets.
- Currently focused on Transformer attention heads and FFN layers, extending to other components like embeddings or layer depths remains future work.
Future Work
Future directions include extending the framework to multi-modal models, integrating reinforcement learning for adaptive sparsity control, and exploring hardware-aware sparse structure design. Additionally, combining this approach with fine-tuning and robustness enhancement techniques could further improve performance and deployment efficiency in real-world scenarios.
AI Executive Summary
Massive language models such as GPT and LLaMA have revolutionized natural language processing but pose significant challenges in deployment due to their enormous size. Traditional compression methods like quantization and low-rank decomposition help but often fall short of balancing efficiency and performance. Structural pruning offers a promising solution by removing redundant network components, yet existing techniques face limitations. Local pruning methods, which prune layer-by-layer, are efficient but ignore global model topology, leading to suboptimal sparsity allocation. Global pruning strategies aim to optimize the entire model's structure but often rely on heuristic importance evaluation or two-stage processes that overlook inter-structure dependencies, resulting in compromised performance.
Guanchen Li and colleagues address these issues with Týr-Pruner, a novel end-to-end global structural pruning framework. The core idea is to construct a supernet that encompasses multiple sparsity configurations across all layers, enabling the simultaneous evaluation of numerous sub-structures. This supernet is built by applying local pruning at various sparsity ratios, guided by Taylor expansion-based importance metrics, which accurately estimate the redundancy of attention heads and FFN neurons. To ensure balanced error propagation during pruning, the authors introduce an expectation error accumulation mechanism, which maintains mutual awareness among different sparse structures.
The optimization process employs an iterative coarse-to-fine search strategy using evolutionary algorithms. Each iteration prunes the model, constructs a narrower supernet with refined sparsity intervals, and searches for the optimal distribution based on a distillation-inspired similarity metric. This approach significantly reduces the search space and accelerates convergence. Experimental results on large models like Llama-3.1-70B demonstrate that Týr-Pruner can remove up to 50% of parameters while retaining 97% of the original performance, outperforming existing state-of-the-art methods such as FLAP and ZipLM. The method requires only 4 million tokens for calibration, making it highly efficient.
This work represents a substantial advancement in model compression, enabling scalable, automated, and high-performance pruning of massive language models. Its implications extend to deploying large models on resource-constrained devices, democratizing AI access, and inspiring future research on end-to-end sparse structure learning. Despite its success, challenges remain in extending to other model components, further reducing computational costs, and enhancing robustness under extreme sparsity. Overall, Týr-Pruner sets a new benchmark for intelligent model compression, with broad potential for industry and academia.
Deep Dive
Abstract
Structural pruning enhances hardware-agnostic inference efficiency for large language models (LLMs) yet often fails to maintain comparable performance. Local pruning performs efficient layer-by-layer compression but ignores global topology. Although global pruning aims to identify an optimal sparse model, intuitive methods typically adopt a two-stage paradigm that first evaluates substructure saliency and then applies global pruning, which ignores inter-structure dependencies and fails to achieve end-to-end optimization. To address these limitations, we propose Týr-the-Pruner, an efficient end-to-end search-based global structural pruning framework. This framework constructs a supernet by repeatedly applying local pruning across a range of sparsity ratios to each layer in an LLM, with the core goal of determining the optimal sparsity distribution under a target overall sparsity ratio. Concretely, we introduce an effective local pruning and an expectation error accumulation approach to improve supernet construction. Furthermore, we employ an iterative prune-and-search strategy with coarse-to-fine sparsity granularity to ensure efficient search convergence. Experimental results show that Týr-the-Pruner achieves state-of-the-art structural pruning, retaining 97% of the dense model's performance while removing a challenging 50% of Llama-3.1-70B's parameters. Code will be available at https://github.com/AMD-AGI/Tyr-the-Pruner.