IPPRO: Importance-based Pruning with PRojective Offset for Magnitude-indifferent Structural Pruning
IPPRO introduces a scale-invariant pruning criterion based on projective geometry, outperforming magnitude-based methods in neural network compression.
Key Findings
Methodology
IPPRO embeds filters into real projective space, defining PROscore as the angular displacement after a single gradient step. This approach eliminates scale dependence by leveraging the properties of projective geometry, where filters are represented as lines rather than vectors, thus ensuring invariance to positive rescaling. The method involves parameter injection to extend the model, enabling gradient-based importance scoring without altering the original architecture. The connection to L0 relaxation theory provides a rigorous foundation for predicting multi-step pruning outcomes from a single gradient step. Extensive experiments across CNNs, Vision Transformers, and LLMs demonstrate the robustness and superiority of IPPRO over existing importance metrics.
Key Results
- On models like ResNet, DeiT, and LLaMA, IPPRO achieves high compression ratios (e.g., 80-90%) with minimal accuracy loss (less than 3%) even without fine-tuning. In ImageNet and Cityscapes datasets, the pruned models maintain performance comparable to unpruned ones, with parameter reductions exceeding 70%. The method consistently outperforms Taylor, VBP, and other gradient-based criteria, especially at high compression levels.
- Theoretical analysis confirms that PROscore aligns with the bifurcation dynamics of L0 relaxation, making it a reliable single-step predictor of multi-step pruning effects. Empirical results show that PROscore rankings are stable across different reparameterizations and activation functions, including non-homogeneous ones like GELU and SiLU.
- In practical scenarios, IPPRO reduces inference latency and computational cost significantly, with experimental latency improvements of up to 1.6× on Vision Transformers, while preserving accuracy. The method also demonstrates robustness in low-data regimes for LLM pruning, requiring only a few calibration samples to produce reliable importance scores.
Significance
This work addresses the fundamental limitation of magnitude-based pruning, which is sensitive to parameter rescaling. By introducing a geometrically grounded, scale-invariant importance measure, IPPRO enhances the robustness and generality of neural network pruning. Its architecture-agnostic nature makes it applicable across diverse models, from CNNs to large language models, facilitating efficient deployment in resource-constrained environments. The theoretical link to L0 relaxation provides a solid foundation for future research, potentially enabling more reliable and theoretically justified pruning strategies. Overall, IPPRO paves the way for more principled neural network compression techniques that are less dependent on ad hoc heuristics.
Technical Contribution
The core innovation lies in translating filter importance assessment into a projective geometric framework, where filters are represented as lines in a space that inherently encodes scale invariance. The PROscore, derived from the angular displacement after a gradient step, provides a robust importance metric unaffected by rescaling. The method's connection to L0 relaxation theory offers a rigorous justification, enabling one-shot importance prediction for multi-step pruning. Additionally, the parameter injection technique ensures practical implementation without significant overhead. These contributions collectively advance the theoretical understanding and practical effectiveness of neural network pruning.
Novelty
This is the first work to embed neural filters into real projective space for importance scoring, leveraging geometric invariance to overcome scale sensitivity. Unlike traditional magnitude or gradient-based metrics, PROscore measures the directional collapse of filters, providing a more intrinsic importance indicator. The theoretical connection to L0 relaxation and the practical parameter injection mechanism further distinguish IPPRO as a novel, principled approach that generalizes across architectures and activation functions.
Limitations
- While PROscore demonstrates robustness across many settings, its effectiveness may diminish with highly non-linear activations or in models with complex normalization schemes where the geometric assumptions are less valid.
- The parameter injection introduces a slight computational overhead, which could be significant in extremely large models or real-time applications.
- Current validation focuses on image classification, segmentation, and language modeling; other tasks like object detection or reinforcement learning require further exploration.
Future Work
Future research will explore adaptive gradient step sizes, stochastic online pruning strategies, and extending the geometric framework to other importance metrics. Theoretical analysis of multi-step dynamics and integration with training-aware pruning methods are also promising directions. Additionally, applying IPPRO to multi-modal models and real-time systems could broaden its impact.
AI Executive Summary
Deep neural networks have achieved remarkable success across various domains, but their increasing size poses significant computational and storage challenges. Traditional importance-based pruning methods predominantly rely on filter magnitude, which is inherently scale-dependent and thus unreliable under parameter reparameterizations. This limitation hampers the robustness and generality of pruning strategies, especially in high compression regimes. To address this, the authors propose IPPRO, a novel importance criterion grounded in projective geometry. By embedding filters into real projective space, IPPRO defines PROscore as the angular displacement of a filter after a single gradient step, capturing its functional importance in a scale-invariant manner. This geometric perspective effectively resolves the singularity at the origin, ensuring that all filters are represented at an equal angular distance from zero, thus making importance assessment independent of magnitude. The connection to L0 relaxation theory further validates that this one-step criterion reliably predicts multi-step pruning dynamics, providing a solid theoretical foundation. Extensive experiments across CNNs, Vision Transformers, and large language models demonstrate that IPPRO consistently outperforms existing importance metrics, especially under high compression ratios and in no-fine-tuning scenarios. The method's architecture-agnostic design and theoretical rigor make it a promising paradigm for neural network compression. Future work aims to extend this framework to online and stochastic pruning, multi-modal models, and real-time deployment, potentially transforming how large models are optimized for resource-constrained environments.
Deep Analysis
Background
随着深度学习模型的不断扩大，模型压缩成为提升推理效率的关键手段。早期方法多依赖滤波器幅值（如L1、L2范数）进行剪枝，简单直观，但受尺度变化影响较大。近年来，梯度敏感性指标如Taylor展开被引入，提升了重要性评估的理论基础，但在非同质激活（GELU、SiLU）中表现不佳。投影几何提供了一种尺度不变的空间，将滤波器表示为线段，避免幅值偏差，成为新研究热点。当前，如何在保证模型性能的同时实现高压缩比，仍是深度学习领域的难点。
Core Problem
传统剪枝方法依赖滤波器幅值，受参数重参数化影响大，导致重要性评估不稳定。梯度指标在非线性激活下表现不佳，缺乏尺度不变性。如何设计一种稳健、普适的尺度不变重要性指标，成为亟待解决的问题。尤其是在高压缩比和无微调场景中，模型性能的保持尤为困难。现有方法缺乏理论支撑，难以在多模型、多任务中推广。
Innovation
引入投影几何，将滤波器映射到实投影空间，定义PROscore作为梯度单步变化的角度偏移，避免幅值尺度依赖。结合参数注入机制，确保模型扩展和计算效率。理论上，PROscore与L0正则化的关系得到严格证明，提供了单步剪枝的理论基础。这些创新使得剪枝方法在不同模型和激活函数下均表现出优异的鲁棒性。该方法突破了传统幅值依赖的限制，为神经网络压缩提供了全新思路。
Methodology
- �� 将滤波器通过同质化映射到实投影空间，消除尺度影响。
- �� 定义PROscore为梯度单步变化后滤波器角度偏移，衡量其重要性。
- �� 通过参数注入机制，将模型扩展为包含额外参数D，用于梯度计算。
- �� 计算PROscore时，利用梯度信息，评估滤波器的方向性变化。
- �� 连接到L0正则化理论，验证单步指标的预测能力。
- �� 在不同模型（CNN、ViT、LLM）中验证效果，确保方法的普适性。
Experiments
在ImageNet、Cityscapes等数据集上，使用ResNet、DeiT、LLaMA模型进行剪枝。比较多种剪枝指标，包括幅值、Taylor、VBP，验证无微调下的性能。采用不同压缩比，评估模型准确率、参数减少和FLOPs变化。还进行不同激活函数和重参数化方案的鲁棒性测试，验证方法的普适性。
Results
IPPRO在高压缩比（如80%以上）下无微调仍能保持模型性能，准确率仅下降2-3%，优于Taylor和VBP。在极端压缩（90%参数减少）时表现尤为优异，模型鲁棒性增强。理论上，PROscore与L0正则化的联系得到验证，确保单步剪枝的可靠性。实验证明该方法在多模型、多任务场景中均优于传统指标。
Applications
该方法适用于模型压缩、边缘设备部署、快速模型剪枝等场景。无需微调即可实现高效剪枝，特别适合资源受限环境。未来可结合在线学习和随机剪枝，推动模型的动态优化和自适应压缩。
Limitations & Outlook
在非同质激活（如GELU）中，PROscore的尺度不变性可能受影响。参数注入机制引入微小计算开销，极大模型中需优化。当前验证主要在图像和语言模型，其他任务如目标检测仍需验证。未来需加强理论分析和算法优化。
Plain Language Accessible to non-experts
想象你在整理一个大型仓库，里面有许多不同大小的箱子。传统方法会根据箱子的大小来决定哪些箱子重要，越大越重要，但这可能误导你，因为有些小箱子装的东西更有价值。IPPRO的方法像是用一个特殊的望远镜，只看箱子里的内容，不管大小，只关注箱子里物品的方向和内容。这样，你可以准确判断哪些箱子可以扔掉，哪些必须留下。它通过数学上的投影，把所有箱子都放在一个公平的角度上，避免被大小迷惑。这样一来，无论箱子怎么变形，都能正确判断出重要的内容，帮助你更高效地整理仓库。
ELI14 Explained like you're 14
想象你在玩一个游戏，你有很多不同大小的宝箱。有的宝箱很大，但里面装的东西其实没什么用；有的宝箱虽然小，但装着非常重要的宝贝。以前的人只看宝箱的大小，觉得越大越重要，但这不总对。IPPRO就像用一种特别的眼镜，只看宝箱里的内容，不管它的大小。它帮你判断哪些宝箱是真的重要，哪些可以扔掉。这个方法用数学把所有宝箱都放在一个公平的角度上，不会被大小迷惑。这样，你就能更快、更准地整理宝箱，节省时间和空间，玩得更开心！
Abstract
Importance-based structured pruning overwhelmingly relies on filter magnitude. This proxy is fundamentally flawed: due to scale invariance, functionally identical filters can receive arbitrarily different importance scores under rescaling. We propose IPPRO (Importance-based Pruning with PROjective Offset), a scale-invariant pruning framework grounded in projective geometry. By embedding filters into real projective space ($\mathbb{RP}^N$), IPPRO resolves the singularity at the origin, placing all filters at an equal angular distance from the zero filter. We define PROscore, which captures functional importance by measuring a filter's angular displacement toward zero under a single gradient step (directional collapse). We further connect PROscore to exact $L_0$ relaxation, proving this one-shot criterion reliably predicts multi-step pruning dynamics. Extensive experiments across CNNs, Vision Transformers, and LLMs (e.g., ResNet, DeiT, LLaMA) demonstrate that IPPRO consistently outperforms existing methods, yielding particularly striking gains under high compression and no-fine-tuning regimes, IPPRO establishes a robust, architecture-agnostic paradigm for neural network compression.