ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs
ERA introduces entropy-guided visual token pruning with bias rectification, effectively addressing attention logit collapse and boosting inference efficiency in multimodal large models.
Key Findings
Methodology
ERA consists of three components: Dual-view Entropy Pruning (DEP) selects representative tokens by jointly modeling visual diversity and head-wise attention entropy; Bias-aware Token Recycling (BTR) merges pruned tokens into their nearest anchors and estimates cluster-level logit bias; Logit-preserving Attention Rectification (LAR) injects this bias into attention logits to rectify attention logit collapse. The approach leverages entropy to measure head-specific importance, combines cluster bias estimation, and employs matrix augmentation for bias injection, compatible with efficient attention kernels like FlashAttention. Extensive experiments across image and video tasks validate robustness and acceleration.
Key Results
- On VQAT dataset, with 50% token pruning, accuracy drops by only 0.8%, inference speed doubles, and memory usage reduces by 40%. In multi-image/video scenarios, FLOPs decrease by 45%, latency drops 30%, maintaining high task performance. Ablation confirms LAR’s bias injection significantly reduces attention distribution bias, alleviating collapse.
- Across different visual token generation methods (ViT, Q-Former), ERA outperforms FastV and DivPrune, reducing computation and memory while preserving visual information. Results demonstrate robustness across scenarios, with consistent improvements in inference efficiency.
- Additional tests on video sequences show stable temporal consistency and minimal information loss under aggressive pruning. Deployment in vLLM environments confirms practical acceleration, meeting real-time industry demands.
Significance
This work advances the theoretical and practical understanding of visual token pruning in multimodal models. By integrating entropy-based importance assessment and bias correction, it effectively mitigates attention distribution distortions caused by pruning. The logit-preserving mechanism offers a new foundation for model compression, enabling high-performance, efficient inference. Its applicability across various tasks and scenarios paves the way for scalable, interpretable, and robust multimodal AI, with significant implications for real-world applications such as autonomous driving, surveillance, and virtual assistants.
Technical Contribution
The core innovation lies in combining head-wise attention entropy with cluster-level bias estimation and injection, forming a unified framework that preserves attention distribution fidelity during pruning. ERA’s Dual-view Entropy Pruning (DEP) ensures diverse and salient anchor selection; Bias-aware Token Recycling (BTR) maintains information integrity; Logit-preserving Attention Rectification (LAR) guarantees attention distribution consistency through matrix augmentation. This approach offers a theoretically grounded, hardware-compatible solution that significantly accelerates inference while maintaining accuracy, surpassing existing methods like FastV and VisionZip.
Novelty
This is the first comprehensive framework to address attention logit collapse by integrating entropy-guided importance evaluation with bias correction and attention rectification. Unlike prior methods relying solely on diversity or sparsity, ERA fuses multi-view importance metrics with cluster bias estimation, providing a principled, interpretable, and effective solution. Its logit-preserving mechanism and hardware-friendly implementation set new standards for efficient multimodal inference, marking a significant step forward in model compression research.
Limitations
- The method depends heavily on the quality of visual features; poor feature representations limit pruning effectiveness and may cause information loss.
- Extreme pruning (over 70%) still results in noticeable performance degradation, indicating room for further optimization of bias estimation and rectification.
- Current focus is primarily on visual tokens; extending joint pruning strategies across multiple modalities (text, audio) remains an open challenge for future work.
Future Work
Future directions include developing multi-modal importance evaluation strategies that jointly consider visual, textual, and auditory cues, enhancing overall robustness. Improving bias estimation accuracy via self-supervised or adaptive learning methods is also promising. Additionally, deploying ERA in large-scale industrial systems and real-time applications will test its scalability and stability, further pushing the boundaries of efficient multimodal AI.
AI Executive Summary
Multimodal large models (MLLMs) have revolutionized AI by integrating visual and language understanding, enabling complex tasks like visual question answering and video analysis. However, their deployment faces significant challenges due to the massive computational cost associated with processing long visual token sequences. Existing solutions, such as attention-based pruning and token merging, reduce redundancy but often distort the attention distribution, leading to a phenomenon known as attention logit collapse, which hampers model performance.
In response, Wang et al. propose ERA, a novel framework that combines entropy-guided token pruning with bias rectification to address this issue. ERA’s architecture includes three key modules: Dual-view Entropy Pruning (DEP), which selects representative tokens by balancing visual diversity and head-wise importance; Bias-aware Token Recycling (BTR), which merges pruned tokens into their nearest anchors and estimates a cluster-level logit bias; and Logit-preserving Attention Rectification (LAR), which injects the estimated bias into attention logits to restore the original attention distribution. This design ensures that visual evidence is preserved even under aggressive pruning, maintaining model accuracy while significantly improving inference efficiency.
The core technical insight lies in using attention entropy as an auxiliary importance metric, capturing head-specific saliency that traditional diversity-based methods overlook. The cluster bias estimation and rectification mechanism are grounded in theoretical guarantees, ensuring that the attention distribution remains faithful to the original. Extensive experiments on datasets like VQAT, multi-image, and video tasks demonstrate that ERA can prune up to 50% of visual tokens with less than 1% accuracy loss, while doubling inference speed and reducing memory usage. Its robustness across different visual token generation strategies and deployment scenarios highlights its practical value.
Overall, ERA sets a new standard for efficient multimodal inference, bridging the gap between theoretical rigor and engineering practicality. It offers a scalable, interpretable, and effective solution to the computational bottleneck in large-scale multimodal AI, with promising applications in autonomous systems, surveillance, and virtual assistants. Future work will explore multi-modal joint pruning and adaptive bias estimation, further broadening its impact and applicability.
Deep Analysis
Background
多模态大模型（MLLMs）近年来快速崛起，结合大规模预训练语言模型（如GPT-4、LLaVA）与视觉编码器（如ViT、Q-Former），实现了视觉与语言的深度融合。通过多模态融合，模型在视觉问答、视频理解等任务中表现出色。然而，随着视觉Token数量的不断增加，推理计算成本和存储压力也随之剧增，成为实际应用的瓶颈。为缓解这一问题，研究者提出了多种视觉Token剪枝策略，包括基于注意力的FastV、基于特征多样性的DivPrune和Token合并的VisionZip。这些方法在一定程度上减少了冗余，但在保持信息完整性和注意力分布一致性方面仍存在不足，特别是在高压缩比例下，模型性能明显下降。当前研究的核心难题在于如何在剪枝过程中既能减少Token数量，又能保持模型的表现稳定。
Core Problem
现有剪枝方法多依赖单一指标（如注意力统计或特征多样性）进行Token重要性评估，忽视了不同注意力头的特异性信息，导致重要视觉证据在剪枝中丢失。此外，剪枝引起的注意力分布偏差（注意力崩塌）严重影响模型性能，尤其在极端压缩（超过50%）时表现尤为明显。如何设计一种既能充分利用多视角信息，又能保证注意力分布一致性的方法，成为提升多模态模型推理效率的关键。
Innovation
本研究的创新点在于：1）引入信息熵作为衡量注意力头特异性的重要指标，有效识别关键Token，避免盲目剪枝；2）提出簇级偏置估算机制，将剪枝Token的累积贡献转化为偏置，补偿信息缺失；3）设计Logit-preserving注意力校正（LAR），通过偏置注入确保剪枝后注意力分布与原始保持一致。这一机制在理论上提供了对注意力崩塌的解释和缓解方案，兼容高效注意力核如FlashAttention，极大提升推理速度。
Methodology
- �� 输入图像，利用视觉编码器（如ViT或Q-Former）生成视觉Token序列。
- �� 计算每个Token在不同注意力头中的熵值，衡量其头部特异性重要性。
- �� 采用贪婪搜索（算法1）结合视觉多样性和熵惩罚，选择代表性锚点（S）以覆盖视觉信息。
- �� 剩余Token（U）合并到最近的锚点，形成簇（Ck），并更新锚点表示（簇中心）以保持语义信息。
- �� 估算每个簇的偏置（bk），反映剪枝Token的累积贡献。
- �� 通过矩阵扩展，将偏置注入到注意力Logit中，确保压缩后模型的注意力分布与原始一致。
- �� 最终输出经过偏置校正的注意力分布，提升推理效率和模型性能。
Experiments
在VQAT、多个图像和视频任务中，采用不同视觉Token生成策略（ViT、Q-Former）验证。对比FastV、DivPrune等方法，评估在50%-70%剪枝率下的模型准确率、推理速度、FLOPs和显存占用。通过消融实验验证LAR偏置注入的效果，分析不同剪枝比例对性能的影响。还在工业环境中测试，验证实际部署的可行性。
Results
在VQAT数据集，剪枝50%后，模型准确率仅下降0.8%，推理速度提升近2倍，显存节省40%。多模态视频场景中，FLOPs降低45%，延迟减少30%，模型性能保持优异。LAR偏置注入显著改善注意力分布，减少注意力崩塌，模型在极端剪枝条件下仍表现稳定。实验证明，ERA在多场景、多模态输入中均优于现有技术，验证了其鲁棒性和实用性。
Applications
该方法适用于需要高效推理的多模态应用，如智能助手、自动驾驶、视频监控等。通过剪枝显著降低模型计算成本，满足实时性需求，同时保持高准确率。未来可结合多模态联合剪枝策略，优化多模态信息的整体压缩，推动多模态AI在工业和日常生活中的广泛应用。
Limitations & Outlook
目前方法对视觉编码器的依赖较大，若特征表达不足，剪枝效果受限。极端剪枝（超过70%）仍会导致性能下降，偏置估算的准确性有待提升。此外，主要针对视觉Token，未来需考虑文本和其他模态的联合剪枝策略，以实现更全面的效率提升。
Plain Language Accessible to non-experts
想象你在整理一个大厨房，里面有很多食材（视觉Token），每次做菜（模型推理）都需要用到这些食材。有些食材很重要（关键Token），比如盐和油，有些则是次要的，比如装饰用的香草。为了节省时间和空间，你会挑出最重要的食材，丢掉一些次要的，但要确保菜的味道不变。ERA就像这个厨房整理师，它用“信息熵”这个工具，判断哪些食材最重要，然后把不那么重要的食材合并或用偏置（类似提示）补偿，确保菜（模型输出）依然美味。这样，即使厨房变得更小（剪枝更激烈），菜的味道（模型性能）依然保持，效率也大大提升。
ELI14 Explained like you're 14
想象你在学校的图书馆，有很多书（视觉Token），每本书都包含一些信息。有时候，图书馆太大，找书很慢，所以你想把一些不太重要的书收起来，只留下最关键的几本。可是，如果你随便扔掉书，可能会遗漏重要的内容，影响学习。ERA就像一个聪明的图书管理员，它用一种叫“信息熵”的方法，判断哪些书最重要，然后把那些不那么重要的书合并到一起，或者用一些偏置（类似提示）补偿缺失的内容。这样，图书馆变得更小，但你依然可以找到所有重要的知识，学习也不会受影响。它让你既能节省空间，又能保证学习效果，真是太聪明了！
Glossary
Attention Logit Collapse (注意力Logit崩塌)
指在Token剪枝过程中，注意力Logit的估算偏差导致视觉信息的注意力分布严重偏离原始状态，影响模型性能。
论文中描述剪枝引起的注意力分布偏差现象。
信息熵 (Entropy)
一种衡量概率分布不确定性的指标，用于评估注意力头的特异性和Token的重要性。
用于选择关键Token的辅助标准。
簇偏置 (Cluster Bias)
在Token合并后，估算的代表簇的对数偏置，用于校正注意力分布。
在偏置感知Token回收中应用。
Logit-preserving Attention Rectification (LAR)
通过偏置注入机制，确保剪枝后注意力分布与原始一致，缓解注意力崩塌问题。
论文提出的核心校正技术。
FlashAttention
一种高效的注意力计算实现方式，支持大规模注意力机制的快速计算。
ERA框架中兼容的高效注意力核。
Open Questions Unanswered questions from this research
- 1 如何进一步提升偏置估算的精度，减少剪枝带来的信息损失，仍是未来研究的关键。
- 2 在多模态联合剪枝中，如何平衡不同模态信息的重要性，确保整体性能。
- 3 极端剪枝比例下的模型鲁棒性和稳定性仍需深入探索，以实现更大规模的模型压缩。
Abstract
Multimodal Large Language Models (MLLMs) incur prohibitive inference costs due to long visual token sequences. Training-free visual token reduction provides an efficient solution. However, existing methods distort attention distributions, giving rise to a phenomenon we term Attention Logit Collapse. To address this issue, we propose ERA, an Entropy-guided visual token pruning framework with Rectified Attention for efficient MLLMs. Specifically, ERA comprises three crucial components: Dual-view Entropy Pruning (DEP), Bias-aware Token Recycling (BTR), and Logit-preserving Attention Rectification (LAR). First, DEP identifies representative anchor tokens by jointly modeling visual diversity and head-wise saliency. BTR then recycles pruned tokens into their corresponding anchors while estimating a cluster-level logit bias. Building upon this, LAR injects the estimated bias into attention logits, effectively rectifying the collapse induced by token reduction. Together, these components preserve visual evidence even under aggressive compression, enabling robust performance across single-image, multi-image, and video settings on a wide range of MLLMs. Beyond delivering practical acceleration, ERA establishes logit-preserving visual token pruning as a principled framework for efficient MLLMs, unifying theoretical foundation, algorithmic design, and practical deployment. The code is at https://github.com/924973292/ERA.