Last But Not Least: Boundary Attention CalibratiON for Multimodal KV Cache Compression
BACON improves multimodal KV-cache compression by 7.5% on average, up to 30.9% under aggressive budgets.
Key Findings
Methodology
BACON calibrates observation window attention with last query attention, suppressing noise and recovering critical visual evidence. It combines intra-layer coherence and inter-layer persistence, offering a plug-and-play KV cache compression mechanism.
Key Results
- On Qwen2-VL-7B with PyramidKV at budget 64, BACON improved DocVQA by 18.7 points.
- On PyramidKV, BACON improved Qwen3-VL-30B-A3B by 12.8 points at budget 64.
- BACON consistently enhances performance across models and compression methods, especially under aggressive budgets.
Significance
BACON significantly enhances inference efficiency for multimodal large language models, reducing memory and latency costs of KV caches, addressing inefficiencies in long visual contexts.
Technical Contribution
BACON introduces a novel KV cache compression method through boundary attention calibration, differentiating from existing observation window-based methods and enhancing visual evidence retention.
Novelty
BACON is the first to utilize last query attention for KV cache compression, calibrating observation window attention and providing a novel evidence recovery mechanism.
Limitations
- In some scenarios, last query attention may introduce noise, affecting compression performance.
- BACON may not completely eliminate noise in extreme cases.
Future Work
Future work could explore BACON's application in more multimodal tasks and further optimize its noise suppression capabilities.
AI Executive Summary
Multimodal large language models excel in vision-language reasoning but suffer from large KV caches and high decoding latency with long visual contexts. Existing compression methods rely on observation window attention for stable token importance estimation, but this aggregation can dilute sparse critical evidence and discard answer-relevant tokens under aggressive compression. BACON calibrates observation window attention with last query attention, suppressing noise and recovering critical visual evidence. Experimental results show that BACON improves multimodal KV-cache compression by 7.5% on average across diverse benchmarks, models, budgets, and compression methods, with gains up to 30.9% under the most aggressive budget. This improvement significantly reduces the memory and latency costs of multimodal large language models, enhancing inference efficiency. BACON's success provides new insights and directions for efficient inference in future multimodal reasoning tasks.
Deep Analysis
Background
Multimodal large language models extend the reasoning and generation capabilities of large language models through visual encoders and projection modules. However, as visual inputs become longer and denser, efficient inference remains challenging. Visual tokens substantially increase prefill length, and the KV cache must retain their representations across layers and attention heads for subsequent decoding. KV cache compression has therefore become an important direction for reducing the memory and latency cost of multimodal large language model inference.
Core Problem
Existing KV cache compression methods typically retain cached tokens according to their estimated importance. A common strategy is to estimate this importance from an observation window, i.e., the final segment of prompt queries, by averaging the attention each cached token receives from these queries. However, this averaging can dilute sparse visual evidence and bias retention away from answer-critical visual tokens.
Innovation
BACON calibrates observation window attention with last query attention to recover diluted visual evidence. • Uses intra-layer coherence and inter-layer persistence to filter noise from last query signals. • Combines observation window attention as a stable retention basis, enhancing visual evidence retention.
Methodology
- �� BACON maintains observation window attention as the stable basis of existing retention scores. • Uses last query attention to calibrate observation window attention, recovering boundary-emergent evidence. • Filters noise from last query signals through intra-layer coherence and inter-layer persistence. • Validates BACON's effectiveness across different models and compression methods.
Experiments
Experiments span multimodal understanding, video reasoning, GUI grounding, and long-context text tasks, using multimodal large language models of different scales and architectures, multiple KV compression methods, and diverse cache budgets.
Results
BACON improves multimodal KV-cache compression by 7.5% on average across diverse benchmarks, models, budgets, and compression methods, with gains up to 30.9% under the most aggressive budget.
Applications
BACON can be used for efficient inference in multimodal large language models, reducing memory and latency costs and enhancing inference efficiency.
Limitations & Outlook
BACON may introduce noise in some scenarios, affecting compression performance. Future work could explore further optimization of its noise suppression capabilities.
Plain Language Accessible to non-experts
Imagine a library with many books (visual tokens), each with varying importance. To save space, we need to decide which books to keep. Existing methods use an observation window (last batch of queries) to decide which books are more important, but this might miss some crucial books. BACON acts like a smart librarian who not only looks at the observation window but also checks the last query to ensure no important books are missed. It also filters out irrelevant books, ensuring that the selected books are the most valuable.
ELI14 Explained like you're 14
Imagine you're playing a game where you need to choose the most important items for your limited backpack space. Existing methods are like only looking at the last level's hints to decide which items are important, but this might miss some key items. BACON is like a smart player who not only looks at the last level's hints but also checks the last hint to make sure no important items are missed. It also filters out irrelevant items, ensuring the selected items are the most valuable.
Glossary
Multimodal Large Language Model (MLLM)
A model that combines visual and language information for reasoning.
Used for handling vision-language tasks.
KV Cache
A cache storing key-value pairs for speeding up inference.
Used in multimodal inference to store visual tokens.
Observation Window Attention
Attention estimation based on the last batch of queries.
Used to estimate token importance.
Last Query Attention
Attention estimation based on the last query.
Used to calibrate observation window attention.
Intra-layer Coherence
Attention consistency among neighboring tokens within the same layer.
Used to filter noise.
Open Questions Unanswered questions from this research
- 1 How to further optimize BACON's compression effect without introducing noise?
- 2 How does BACON perform in more complex multimodal tasks?
Applications
Immediate Applications
Multimodal Inference
BACON can be used to improve inference efficiency in multimodal large language models, reducing memory and latency costs.
Long-term Vision
Intelligent Information Retrieval
By improving multimodal inference, BACON can be used for more intelligent information retrieval systems.
Abstract
Multimodal Large Language Models (MLLMs) achieve strong vision-language reasoning but incur large KV caches and high decoding latency with long visual contexts. Existing compression methods rely on observation window attention for stable token importance estimation, yet this aggregation can dilute sparse critical evidence and discard answer-relevant tokens under aggressive compression. We identify last query attention as a complementary signal for recovering such evidence, though its irrelevant signals may introduce additional noise. We propose BACON, a plug-and-play method that calibrates observation window attention with last query evidence while suppressing noise through intra-layer coherence and inter-layer persistence. Across diverse benchmarks, models, budgets, and compression methods, BACON improves multimodal KV-cache compression by 7.5% on average under the most aggressive budget, with gains up to 30.9%.