Multi-Dimensional Pruning: Joint Channel, Layer and Block Pruning with Latency Constraint
Joint channel, layer, and block pruning via MINLP with accurate latency modeling achieves high pruning ratios and improved speed/accuracy trade-offs.
Key Findings
Methodology
This paper introduces Multi-Dimensional Pruning (MDP), which unifies channel, layer, and block pruning through a comprehensive optimization framework. It employs Taylor importance scores to evaluate per-layer importance, constructs latency cost matrices based on hardware measurements, and models the pruning problem as a MINLP. The approach groups layers within blocks to handle joint decisions, enabling the solver to identify the globally optimal sparse structure under latency constraints. The process involves computing importance scores, building latency matrices, formulating the MINLP, solving with Pyomo and MindtPy, and fine-tuning the resulting model. This integrated method effectively balances high pruning ratios with minimal accuracy loss, demonstrating superior performance across multiple tasks.
Key Results
- On ImageNet classification, ResNet50 pruned by 85% achieved 70.0% Top-1 accuracy, with FPS reaching 5262 images/sec, surpassing prior methods like HALP (68.6%, 4101 im/s) by significant margins. The method maintained high accuracy at large pruning ratios.
- In 3D detection, pruning StreamPETR at 45% ratio increased FPS from 31.7 to 37.3 and improved mAP from 0.449 to 0.451, outperforming dense baselines. Larger ratios (70%) also yielded notable speed and accuracy gains.
- Across tasks, the framework consistently outperformed existing approaches, especially at high pruning ratios, confirming its robustness and generalization.
Significance
This work addresses the critical challenge of model efficiency for deployment on resource-constrained hardware. By integrating multi-dimensional pruning with precise latency modeling, it enables aggressive compression without sacrificing accuracy. The approach advances the state-of-the-art in hardware-aware model compression, facilitating real-time applications like autonomous driving and robotics. It bridges the gap between theoretical optimization and practical deployment, providing a scalable solution adaptable to various architectures and hardware platforms. The methodology’s ability to achieve high pruning ratios while maintaining performance marks a significant step forward for efficient deep learning.
Technical Contribution
The main technical innovations include:• A unified framework that combines channel, layer, and block pruning within a single optimization problem.• A hardware-informed latency model that captures the impact of both input and output channel variations, enabling precise latency constraints.• Reformulation of pruning as a MINLP, solved efficiently with Pyomo and MindtPy, to obtain globally optimal sparse structures in one pass.• Use of importance scores (Taylor) integrated with latency models to guide pruning decisions, balancing speed and accuracy.• Demonstration of scalability across multiple tasks and architectures, with superior results compared to prior heuristics and local optimization methods. This work opens new avenues for principled, hardware-aware model compression.
Novelty
This research is the first to unify channel, layer, and block pruning into a single, globally optimized framework based on MINLP, incorporating a detailed hardware latency model that considers both input and output channels simultaneously. Unlike prior methods that rely on heuristic or layer-only models, this approach ensures optimality at high pruning ratios. Its block grouping strategy and single-pass solution significantly improve efficiency and effectiveness, setting a new benchmark in hardware-aware pruning. The integration of these elements constitutes a novel contribution to model compression literature.
Limitations
- The approach depends heavily on the accuracy of the hardware latency lookup table; discrepancies between measured and actual latency may affect optimality.
- Solving MINLPs can be computationally intensive, especially for very large models or extremely high pruning ratios, potentially limiting real-time applicability.
- Fine-tuning after pruning remains necessary, adding additional computational overhead, which may be challenging in resource-limited scenarios.
Future Work
Future directions include developing faster MINLP solvers or approximation algorithms to reduce computational costs, extending the latency model to dynamic or adaptive hardware environments, and integrating automated neural architecture search for even more optimal pruning strategies. Additionally, expanding the framework to multi-task learning and multi-modal models could further broaden its impact.
AI Executive Summary
Deep neural networks have revolutionized computer vision, yet their increasing size hampers deployment on edge devices due to latency and resource constraints. Traditional pruning methods, such as channel pruning, are limited to removing redundant filters but struggle with high pruning ratios necessary for real-time applications. Moreover, existing hardware-aware approaches often rely on simplified latency models that ignore input channel variations, leading to suboptimal solutions. To address these issues, this work introduces a comprehensive Multi-Dimensional Pruning (MDP) framework that jointly optimizes channel, layer, and block pruning decisions through a unified MINLP formulation. Central to this approach is a hardware-informed latency model that captures the combined effects of input and output channel variations, enabling precise latency constraints. The process begins with importance score computation using Taylor metrics, followed by constructing latency matrices based on hardware measurements. Layers are grouped within blocks to facilitate joint decision-making, ensuring structural integrity. The formulated MINLP is solved efficiently using Pyomo and MindtPy, yielding an optimal sparse architecture in a single pass. Extensive experiments across multiple tasks demonstrate the method’s effectiveness: on ImageNet, ResNet50 achieves 70.0% Top-1 accuracy at 5262im/s FPS after 85% pruning, outperforming prior art; on NuScenes, StreamPETR’s mAP improves from 0.449 to 0.451 at 45% pruning, with FPS rising from 31.7 to 37.3. These results confirm that the proposed framework significantly advances the state-of-the-art in hardware-aware model compression, especially at high pruning ratios. The approach’s scalability and robustness suggest broad applicability in real-world scenarios, from autonomous vehicles to robotics. Future work will focus on reducing computational complexity, adapting to dynamic hardware environments, and extending to multi-task models, further pushing the boundaries of efficient deep learning deployment.
Deep Dive
Abstract
As we push the boundaries of performance in various vision tasks, the models grow in size correspondingly. To keep up with this growth, we need very aggressive pruning techniques for efficient inference and deployment on edge devices. Existing pruning approaches are limited to channel pruning and struggle with aggressive parameter reductions. In this paper, we propose a novel multi-dimensional pruning framework that jointly optimizes pruning across channels, layers, and blocks while adhering to latency constraints. We develop a latency modeling technique that accurately captures model-wide latency variations during pruning, which is crucial for achieving an optimal latency-accuracy trade-offs at high pruning ratio. We reformulate pruning as a Mixed-Integer Nonlinear Program (MINLP) to efficiently determine the optimal pruned structure with only a single pass. Our extensive results demonstrate substantial improvements over previous methods, particularly at large pruning ratios. In classification, our method significantly outperforms prior art HALP with a Top-1 accuracy of 70.0(v.s. 68.6) and an FPS of 5262 im/s(v.s. 4101 im/s). In 3D object detection, we establish a new state-of-the-art by pruning StreamPETR at a 45% pruning ratio, achieving higher FPS (37.3 vs. 31.7) and mAP (0.451 vs. 0.449) than the dense baseline.