PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference
PACE achieves 93.8% performance retention while reducing visual tokens by 90%, enabling a 3.1x speedup via pixel compression and dual-attention extraction.
Key Findings
Methodology
PACE introduces a training-free inference framework with two stages: the Condense stage uses an Adaptive Pixel Compressor (APC) to downsample redundant inputs, and the Extract stage employs a Dynamic Dual-Attention Extractor (DDAE) to retain critical tokens by fusing visual and semantic signals.
Key Results
- On Qwen2.5-VL-7B, PACE retains 93.8% of original performance while using only 10% of visual tokens, achieving a 3.1x speedup in TTFT.
- PACE achieves 70.90% accuracy on OCRBench at 10% token budget, outperforming the best baseline by 17.90 percentage points.
- PACE achieves 48.94% ANLS on DocVQA at 5% token budget, significantly outperforming other methods.
Significance
PACE addresses the dual bottlenecks in vision-language inference: the vision encoder and LLM prefill stages. By compressing visual data before encoding and extracting key tokens after encoding, PACE significantly reduces computational costs while preserving both global context and fine-grained details. This framework is particularly impactful for real-time, high-resolution tasks.
Technical Contribution
Key contributions include: 1) the Adaptive Pixel Compressor (APC) for pre-encoder input resolution adjustment; 2) the Dynamic Dual-Attention Extractor (DDAE) for fusing visual and semantic signals; 3) a training-free, plug-and-play framework compatible with existing token pruning methods.
Novelty
PACE is the first framework to unify pre-encoder compression and post-encoder extraction for vision-language inference. Its dual-attention mechanism outperforms existing methods in preserving both global layouts and fine-grained details.
Limitations
- Performance drops under extreme token budgets (e.g., 5%) due to detail loss, especially in layout-sensitive tasks.
- APC's pixel compression may affect global context in complex backgrounds.
- The framework does not optimize autoregressive decoding, limiting end-to-end inference speed.
Future Work
Future work could explore integrating PACE with autoregressive decoding optimizations for further efficiency gains and extending its applicability to complex multimodal tasks like video inference.
AI Executive Summary
Vision-language models (VLMs) excel at multimodal reasoning but face escalating inference costs due to the proliferation of visual tokens in high-resolution inputs. Existing token pruning methods focus on post-encoder stages, neglecting the substantial computational overhead of the vision encoder. Moreover, these methods often fail to preserve both global context and fine-grained details under strict token budgets.
To address these challenges, PACE (Pixel-Adaptive Condense and Extract) introduces a training-free framework with two stages. The Condense stage employs an Adaptive Pixel Compressor (APC) to dynamically downsample redundant inputs, while the Extract stage uses a Dynamic Dual-Attention Extractor (DDAE) to retain critical tokens by fusing visual and semantic signals. Experiments on Qwen2.5-VL-7B show that PACE retains 93.8% of original performance while using only 10% of visual tokens, achieving a 3.1x speedup in time to first token (TTFT).
PACE significantly reduces inference costs while excelling in detail-sensitive tasks like OCRBench and DocVQA. However, challenges remain in extreme compression scenarios and unoptimized autoregressive decoding. PACE offers a promising direction for efficient multimodal inference and lays the groundwork for future research.
Deep Analysis
Background
Vision-language models have advanced significantly, integrating visual perception with linguistic reasoning. However, high-resolution inputs lead to a rapid increase in visual tokens, causing inference costs to skyrocket. Existing methods like FastV and VisionZip focus on pruning visual tokens but overlook the vision encoder's computational overhead.
Core Problem
The core problem is the dual bottleneck in high-resolution inference: the vision encoder processes redundant pixels, and the language model handles lengthy token sequences, resulting in inefficiency.
Innovation
PACE's key innovations include: 1) Adaptive Pixel Compressor (APC) for dynamic input resolution adjustment; 2) Dynamic Dual-Attention Extractor (DDAE) for fusing visual and semantic signals; 3) a training-free, plug-and-play framework compatible with existing pruning methods.
Methodology
- �� Condense stage: APC computes global redundancy and local detail contrast via lightweight feature previews to dynamically adjust input resolution.
- �� Extract stage: DDAE fuses visual self-attention and semantic cross-attention, dynamically weighting them to select critical tokens.
- �� Integration: PACE is seamlessly integrated into existing VLMs like Qwen2.5-VL.
Experiments
Experiments were conducted on Qwen2.5-VL-7B using nine datasets (e.g., TextVQA, DocVQA). Baselines included FastV, SparseVLM, and others. Both fixed-resolution and dynamic-resolution settings were evaluated.
Results
PACE retains 93.8% of performance at 10% token budget, achieving a 3.1x TTFT speedup. On OCRBench, it achieves 70.90% accuracy, outperforming the best baseline by 17.90 percentage points.
Applications
PACE is suitable for scenarios requiring efficient inference, such as real-time image question answering, document analysis, and multimodal search. Its training-free nature facilitates integration into existing systems.
Limitations & Outlook
PACE may lose critical details under extreme token budgets, affecting performance in layout-sensitive tasks. Additionally, it does not optimize autoregressive decoding, limiting its end-to-end speed.
Plain Language Accessible to non-experts
Imagine PACE as a smart library system. The library has thousands of books (pixels and tokens), but only a few are relevant to your research. First, a librarian (APC) scans the shelves and removes unnecessary books, keeping the important ones neatly organized (Condense stage). Then, another librarian (DDAE) picks the most relevant books based on your specific research topic (Extract stage). This way, you get what you need quickly without sifting through irrelevant material.
ELI14 Explained like you're 14
Think of PACE like a video game helper! You have a huge map to explore, but limited time. PACE is your smart guide. First, it shrinks the map to only show the important areas (Condense stage). Then, it highlights the exact spots you need to check out for your mission (Extract stage). Cool, right? This way, you save time and still find all the treasures you need!
Glossary
Vision-Language Model
A deep learning model that integrates visual and linguistic information for reasoning.
PACE optimizes inference for such models.
Adaptive Pixel Compressor
A module that dynamically adjusts input resolution.
Used in PACE's Condense stage.
Dynamic Dual-Attention Extractor
A module that fuses visual and semantic signals to select critical tokens.
Used in PACE's Extract stage.
Token Budget
A constraint on the number of visual tokens used.
Used to evaluate PACE's performance under compression.
TTFT (Time to First Token)
The time taken to generate the first output token from input.
PACE significantly reduces TTFT.
Open Questions Unanswered questions from this research
- 1 How can detail loss under extreme token budgets be minimized?
- 2 How can PACE be extended to video or 3D data inference tasks?
- 3 What optimizations can be made for autoregressive decoding to improve end-to-end speed?
Applications
Immediate Applications
Real-time Image QA
Improves response speed for real-time image question-answering systems, useful for search engines and customer support.
Document Analysis
Efficiently processes visual and textual data in documents for legal, financial, and automated workflows.
Long-term Vision
General Multimodal Inference
PACE could evolve into a universal framework for complex tasks like video analysis and augmented reality.
Abstract
Vision-Language Models (VLMs) demonstrate exceptional visual reasoning capabilities, yet their inference costs escalate rapidly with the proliferation of visual tokens. Existing visual token pruning methods exhibit two fundamental limitations. First, most approaches operate exclusively post-vision encoder, leaving the substantial latency of the visual encoding phase unoptimized. Second, under strict token budgets, these methods often fail to jointly preserve holistic visual contexts and fine-grained details, leading to performance degradation. To address these bottlenecks, we propose PACE (Pixel-Adaptive Condense and Extract), a training-free inference framework that accelerates both the vision encoder and the Large Language Model (LLM) via a unified Condense-and-Extract paradigm. During the Condense stage, an Adaptive Pixel Compressor (APC) evaluates visual information density prior to encoding, adaptively downsampling redundant inputs, curtailing encoder computation while preserving global context and essential visual cues. In the Extract stage, a Dynamic Dual-Attention Extractor (DDAE) selectively retains visual tokens via a fusion of internal visual signals from the encoder and semantic signals from the LLM, safeguarding task-critical details. By integrating PACE into Qwen2.5-VL-7B, the model retains 93.8% of its original performance while utilizing only 10% of the visual tokens, yielding a 3.1x speedup in time to first token (TTFT). Our code is available at https://github.com/jjL357/PACE.