CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuning
CLOVER leverages cross-layer SVD-based orthogonal vectors for pruning and fine-tuning, achieving high compression with minimal performance loss.
Key Findings
Methodology
CLOVER applies SVD to each attention head’s Q-K and V-O pairs, decomposing them into orthogonal bases. By truncating singular values, it enables high-ratio pruning; alternatively, these values serve as trainable parameters for full-rank fine-tuning. This approach avoids additional transformation matrices, maintaining parameter count. Extensive experiments on GPT-2 XL, LLaMA-3.2-11B, and other models demonstrate superior pruning efficiency and performance recovery, outperforming SOTA methods like LoRA and PiSSA. The core innovation is the cross-layer orthogonalization, which reduces redundancy and enhances model compressibility without parameter overhead.
Key Results
- Pruning 70% of Q-K pairs in GPT-2 XL with CLOVER results in perplexity comparable to pruning only 8% with vanilla methods, showing a 8.75× efficiency gain.
- Fine-tuning singular values in LLaMA-2 7B improves performance by 7.6% over LoRA and DoRA across eight commonsense reasoning tasks.
- In multimodal models like DeepSeek-V2-Lite and Whisper, CLOVER achieves high pruning ratios (>50%) while maintaining near-original performance, demonstrating broad applicability.
Significance
This work addresses the critical challenge of reducing memory and computational burdens in large models without performance degradation. By introducing a parameter-efficient, orthogonal vector-based pruning and fine-tuning framework, CLOVER paves the way for deploying massive models in resource-constrained environments. Its ability to preserve model accuracy while significantly compressing parameters has profound implications for AI deployment, especially on edge devices and in real-time applications. Theoretically, it advances understanding of low-rank structures in attention mechanisms, offering a new paradigm for model compression.
Technical Contribution
CLOVER’s main contribution is the innovative use of SVD on attention head matrices, treating Q-K and V-O pairs as low-rank matrices. It generates orthogonal bases without additional transformation matrices, drastically reducing parameter overhead. The singular values serve as a compact, trainable set of parameters for pruning and full-rank fine-tuning, enabling high compression ratios. This approach is compatible with various models and tasks, outperforming existing methods like LoRA, PiSSA, and SVFT in both pruning ratio and performance recovery. It also provides a theoretical guarantee of orthogonality and low-rank approximation, enhancing model interpretability and robustness.
Novelty
This is the first work to leverage cross-layer low-rank decomposition of attention matrices via SVD for both pruning and fine-tuning without adding extra parameters. Unlike prior methods that rely on transformation matrices or fixed orthogonal bases, CLOVER dynamically learns orthogonal bases through singular values, enabling efficient linear combinations. Its unique approach of treating Q-K and V-O pairs as low-rank matrices within each attention head introduces a new perspective on model compression, combining theoretical rigor with practical efficiency.
Limitations
- While CLOVER achieves high pruning ratios, extremely aggressive pruning can still lead to performance degradation, especially in tasks requiring fine-grained attention. The method also depends on the presence of linear redundancy, which varies across models.
- The approach is primarily designed for attention-based architectures; its effectiveness on non-attention models remains to be explored.
- Fine-tuning singular values, although parameter-efficient, still requires computational resources, limiting real-time deployment in some scenarios.
Future Work
Future research could focus on adaptive singular value selection, integrating CLOVER with quantization and sparsity techniques for even higher compression. Exploring its applicability to non-attention architectures and dynamic pruning strategies could further broaden its impact. Additionally, developing theoretical bounds for performance loss under extreme compression and extending the approach to continual learning settings are promising directions.
AI Executive Summary
The rapid growth of large-scale pre-trained models has revolutionized AI, but their immense size poses significant challenges for deployment, especially in resource-constrained environments. Traditional pruning methods often compromise performance when aggressively reducing parameters, due to the complex redundancy within attention mechanisms. Addressing this, CLOVER introduces a novel framework that leverages cross-layer singular value decomposition (SVD) to orthogonalize attention vectors efficiently.
By decomposing the Q-K and V-O pairs within each attention head into low-rank orthogonal bases, CLOVER enables high-ratio pruning and parameter-efficient fine-tuning without increasing model size. This approach circumvents the need for additional transformation matrices, maintaining the original parameter count while significantly reducing redundancy. Extensive experiments on models like GPT-2 XL and LLaMA-2 demonstrate that CLOVER can prune up to 70% of attention vectors with minimal perplexity increase, outperforming vanilla and state-of-the-art methods.
Furthermore, fine-tuning the singular values allows the model to recover and even surpass baseline performance, achieving an average improvement of 7.6% across multiple tasks. The method’s broad applicability to various architectures, including multimodal models, underscores its versatility. The core innovation lies in the cross-layer orthogonalization strategy, which offers a new paradigm for model compression—balancing efficiency, performance, and interpretability.
Looking ahead, integrating CLOVER with quantization, sparsity, and dynamic pruning techniques promises even greater compression ratios. Its potential to enable large models to run efficiently on edge devices could transform AI deployment, making powerful models accessible in real-world, low-resource settings. Despite some limitations at extreme pruning levels, CLOVER sets a new standard for parameter-efficient model compression, opening exciting avenues for future research and application.
Deep Dive
Plain Language Accessible to non-experts
想象你在整理一个超级大的工具箱，里面装满了各种工具。有些工具其实是重复的或者用得很少。CLOVER就像用一种聪明的方法，把这些工具分类成几组最核心的工具，只保留最重要的几组。那些不重要的工具可以安全地扔掉，不会影响你完成大部分工作。这样，工具箱变得更轻便、更整洁，还能更快找到需要的工具。通过这种方式，模型也变得更小、更快，存储空间和计算资源都大大减少，就像你用更少的工具完成了同样的任务。
ELI14 Explained like you're 14
想象你有一个超级大的书架，上面堆满了各种书。有些书其实是重复的或者用得很少。要让书架变得更整洁，你可以把那些重复或不用的书拿掉，只留下最重要的几本。可是，如果你只随便扔掉一些书，可能会丢掉一些重要的内容。CLOVER就像用一种聪明的办法，把所有书按照主题分类成几组最核心的书本，其他的可以安全地去掉。这样，书架变得更轻便，找书也更快了。它还可以帮你把剩下的书重新整理，让它们变得更有用、更容易理解。就像这样，模型也可以用这种方法变得更小、更快，还能保持原来的功能。
Abstract
Decoder-only models generate tokens autoregressively by caching key/value vectors, but as the cache grows, inference becomes memory-bound. To address this issue, we introduce CLOVER (Cross-Layer Orthogonal Vectors), a novel approach that treats pairs of attention layers as a set of low-rank decompositions. CLOVER applies Singular Value Decomposition (SVD) to the \( Q \)-\( K \) and \( V \)-\( O \) pairs within each attention head. The resulting singular values can either guide pruning or serve as trainable parameters for efficient fine-tuning of all orthogonal vectors. After pruning or fine-tuning, these values are reintegrated into the model without increasing its parameter count. We apply CLOVER to various models, including GPT-2 XL, DeepSeek-V2-Lite, Whisper-Large-v3, Stable Diffusion XL, and LLaMA-3.2-11B-Vision. Our results demonstrate that CLOVER significantly improves pruning efficiency. For instance, the perplexity of pruning 70\% of the \( Q \)-\( K \) pairs in GPT-2 XL is similar to that of pruning just 8\% with vanilla methods. Fine-tuning the singular values further results in a full-rank update, outperforming state-of-the-art methods (LoRA, DoRA, HiRA, and PiSSA) by 7.6\%, 5.5\%, 3.8\%, and 0.7\%, respectively, on eight commonsense tasks for LLaMA-2 7B.