CONDUIT: A Unified Residual-Stream Restoration Framework for KV Cache Reuse in Vision-Language Models
CONDUIT framework achieves 97.0-99.5% performance at 10% refresh budget.
Key Findings
Methodology
The CONDUIT framework implements KV cache reuse through residual-stream restoration. It ranks cached visual tokens using cached-key query attention and a cached-value-norm proxy, applying empirical image-level relevance amplification before global selection. This method is training-free, preserving model architecture and adding only a query-conditioned scoring pass at inference.
Key Results
- At a 10% refresh budget, CONDUIT achieves 97.0-99.5% performance across five datasets, improving by about 1.7 points over ProphetKV on average.
- On the MMLongBench-Doc latency subset, it uses 13.5% of full-prefill FLOPs and achieves a 2.99x time-to-first-token speedup.
- Across single- and multi-image settings, CONDUIT preserves most of full-prefill quality with significantly less prefill compute.
Significance
The CONDUIT framework enables efficient KV cache reuse in vision-language models, significantly reducing computational costs. By employing a training-free refresh policy, it enhances model response speed and computational efficiency without altering the model architecture, addressing the issue of cache invalidation due to visual prefix changes.
Technical Contribution
CONDUIT achieves residual-stream restoration for visual KV cache by combining cached-key query attention with cached-value norm proxy. This method provides a training-free refresh strategy that significantly enhances cache reuse efficiency without altering model architecture and weights.
Novelty
CONDUIT is the first to frame visual KV cache reuse as a residual-stream restoration problem, proposing a norm-weighted attention-based refresh strategy, distinct from traditional attention selection methods.
Limitations
- In high-complexity scenarios, a higher refresh budget may be needed to maintain performance.
- Handling of multi-image prompts may lack precision in certain cases.
Future Work
Future work could explore applicability across different vision-language model architectures and performance on larger-scale datasets.
AI Executive Summary
Vision-language models (VLMs) often face the challenge of re-encoding expensive visual prefixes when dealing with recurring visual content. Existing methods fail to efficiently reuse caches when prefixes change, leading to wasted computational resources.
The CONDUIT framework implements KV cache reuse through residual-stream restoration, using norm-weighted attention for ranking and global selection during inference. This method is training-free, preserving model architecture and adding only a query-conditioned scoring pass.
In experiments, CONDUIT achieves 97.0-99.5% performance at a 10% refresh budget, significantly reducing computational costs and enhancing cache reuse efficiency under multi-image prompts.
Deep Analysis
Background
Vision-language models answer questions grounded in natural images, document pages, and user interfaces. As visual content recurs, re-encoding visual prefixes becomes costly. Existing methods fail to efficiently reuse caches when prefixes change, leading to wasted computational resources.
Core Problem
The problem of cache invalidation due to visual prefix changes. Existing methods fail to efficiently reuse caches when prefixes change, leading to wasted computational resources.
Innovation
The CONDUIT framework implements KV cache reuse through residual-stream restoration, using norm-weighted attention for ranking and global selection during inference.
Methodology
- �� Rank cached visual tokens using cached-key query attention and cached-value norm proxy.
- �� Apply empirical image-level relevance amplification.
- �� Perform global selection during inference, preserving model architecture.
Experiments
Experiments conducted on five datasets: LongDocURL, MMLongBench-Doc, SlideVQA, InfoSeek, and ViQuAE. Tested on three VLM backbones.
Results
At a 10% refresh budget, CONDUIT achieves 97.0-99.5% performance across five datasets. On the MMLongBench-Doc latency subset, it uses 13.5% of full-prefill FLOPs and achieves a 2.99x time-to-first-token speedup.
Applications
Applicable in scenarios requiring efficient handling of recurring visual content, such as document retrieval and multi-image reasoning.
Limitations & Outlook
In high-complexity scenarios, a higher refresh budget may be needed to maintain performance. Handling of multi-image prompts may lack precision in certain cases.
Plain Language Accessible to non-experts
Imagine you're in a library, and the librarian has a notebook with keywords for each book. When someone asks a question, the librarian can quickly look up the notebook instead of flipping through each book. This is like the CONDUIT framework, which speeds up the query process by caching keywords. Even if the question's beginning changes, as long as the keywords remain the same, the librarian can quickly find the answer.
ELI14 Explained like you're 14
Imagine you're playing a game with many levels, each with different challenges. You can save your game progress, so even if you switch to a new level, you can quickly load previous progress without starting over. This is like the CONDUIT framework, helping models quickly reuse previous computation results when handling new questions, saving time and resources!
Glossary
Vision-Language Model
A model that combines visual and language information for tasks like image captioning or question answering.
Used for joint tasks involving natural images and text.
KV Cache
A caching mechanism that stores key-value pairs to speed up subsequent queries.
Used in vision-language models to store visual prefix information.
Residual Stream
A shared channel in a model for transmitting information, where attention layers read from and write to.
Used to explain cache reuse mechanism in CONDUIT.
Norm-Weighted Attention
An attention mechanism combining attention weights and norm information to assess importance.
Used in CONDUIT's refresh strategy.
Refresh Budget
The number of visual tokens allowed to be recomputed during cache reuse.
Used to control CONDUIT's computational cost.
Open Questions Unanswered questions from this research
- 1 How to achieve efficient KV cache reuse on larger-scale datasets?
- 2 How to further optimize cache reuse strategy under multi-image prompts?
Applications
Immediate Applications
Document Retrieval
Accelerates document retrieval by quickly reusing caches, improving query efficiency.
Long-term Vision
Intelligent Assistants
Enhances response speed and accuracy in intelligent assistants during multi-turn conversations.
Abstract
Vision-language models (VLMs) often answer new questions about recurring visual content, where reusing the key-value (KV) cache can avoid re-encoding expensive visual prefixes. Exact-prefix reuse, however, fails when the same visual content appears under a changed prefix. Selective recomputation can recover quality under a small visual-token budget, but only when the right stale tokens are refreshed. Raw-attention selection can waste budget on high-attention tokens with small value-norm proxy scores and on query-irrelevant images. To address these failure modes, we propose CONDUIT, a training-free refresh policy that unifies single- and multi-image reuse as residual-stream restoration. Building on norm-weighted attention, CONDUIT ranks cached visual tokens using cached-key query attention and an accessible pre-output cached-value-norm proxy, then applies empirical image-level relevance amplification before one global selection. With one image, the coefficient is one and the rule reduces to intra-image token selection. The method preserves model architecture and weights, adding only a single query-conditioned scoring pass at inference. At a 10% refresh budget, CONDUIT achieves 97.0-99.5% of the corresponding full-prefill five-dataset average across three VLM backbones and leads budgeted methods on average; on the MMLongBench-Doc latency subset, it uses 13.5% of full-prefill FLOPs and achieves a 2.99x time-to-first-token speedup.