DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruning
DocPruner employs adaptive patch pruning based on global token attention, reducing storage by 50-60% with minimal performance loss.
Key Findings
Methodology
The proposed framework leverages attention scores from the final Transformer layer, specifically the global token, to assess patch importance. It computes mean and standard deviation of attention-based importance scores, then dynamically sets a threshold for pruning. This adaptive process ensures critical patches are retained across diverse document types without retraining models. The core algorithm involves extracting global token attention, calculating statistical features, and applying a threshold to prune less informative patches, grounded in information bottleneck theory for optimal information retention.
Key Results
- On datasets ViDoRe-V2 and JinaVDR-Bench, DocPruner achieves 50-60% patch pruning with negligible nDCG@5 performance drop (e.g., ColQwen2.5 drops 0.0038 from 0.5508 to 0.5470 at 51.6% pruning). Similar results are observed across models, demonstrating robustness.
- Compared to fixed-threshold and non-adaptive methods, adaptive pruning maintains higher retrieval accuracy at comparable compression ratios. Ablation studies confirm the importance of global token attention and entropy-based thresholds in preserving semantic content.
- The approach generalizes well across multilingual and complex layout documents, significantly reducing storage costs while maintaining state-of-the-art retrieval performance.
Significance
This work addresses the critical bottleneck of storage overhead in multi-vector visual document retrieval systems. By enabling substantial compression without performance loss, it facilitates large-scale deployment in industry, reducing hardware costs and improving retrieval speed. The theoretical integration of information bottleneck principles offers a solid foundation for future feature selection and compression strategies in multimodal retrieval, advancing both academic research and practical applications.
Technical Contribution
The paper introduces a novel, query-agnostic, adaptive patch pruning method based on global token attention scores, combined with an information-theoretic thresholding mechanism. It bridges the gap between high-fidelity multi-vector representations and storage efficiency, providing a scalable, model-agnostic solution. Theoretically, it grounds the pruning strategy in mutual information maximization, ensuring critical semantic information is preserved while reducing redundancy.
Novelty
This is the first work to utilize global token attention for adaptive patch importance estimation in visual document retrieval. Unlike prior fixed or heuristic pruning methods, it dynamically adjusts pruning ratios based on document content, guided by information bottleneck theory, offering a significant step forward in efficient multimodal retrieval.
Limitations
- The method relies on Transformer-based attention mechanisms, which may limit applicability to models without such structures. Its effectiveness in non-transformer architectures remains untested.
- Parameter k for threshold adjustment requires manual tuning, potentially affecting performance across different datasets or document types.
- Extremely sparse or dense documents might still pose challenges, leading to over-pruning or under-pruning, affecting retrieval accuracy.
Future Work
Future research could explore end-to-end learning of adaptive thresholds, integrating the pruning mechanism into training. Extending the approach to other model architectures and modalities, such as video or 3D data, is also promising. Additionally, real-time adaptive pruning for dynamic document streams and multi-modal fusion strategies will further enhance practical deployment.
AI Executive Summary
Visual document retrieval (VDR) has become a vital tool in information management, especially with the advent of large vision-language models (LVLMs). These models often employ multi-vector representations, dividing each document into numerous patch embeddings to capture fine-grained details. While this approach significantly improves retrieval accuracy, it introduces a severe storage challenge: hundreds to thousands of vectors per page lead to prohibitive costs in large-scale systems. Addressing this, the paper introduces DocPruner, a novel framework that employs an adaptive pruning strategy based on attention scores from the Transformer’s global token. This method assesses the importance of each patch without requiring retraining, dynamically adjusting the pruning ratio according to document content. The core innovation lies in leveraging the attention distribution to identify and discard redundant patches, guided by an information bottleneck principle that ensures critical information is preserved. Extensive experiments across multiple datasets and models demonstrate that DocPruner can reduce storage by 50-60% with negligible performance loss, outperforming fixed-threshold and non-adaptive methods. Its robustness across multilingual and diverse document types underscores its potential for real-world deployment. This work not only alleviates the storage bottleneck but also provides a theoretical foundation for adaptive feature selection in multimodal retrieval, paving the way for more scalable and efficient systems. Future directions include integrating end-to-end learning of pruning parameters, extending to other modalities, and optimizing for real-time applications, promising a significant leap forward in large-scale visual document retrieval technology.
Deep Analysis
Background
随着数字化信息的爆炸式增长，视觉文档在商务、教育和科研中的应用日益普及。早期方法依赖光学字符识别（OCR）提取文本，存在布局信息丢失和误差累积的问题。近年来，深度学习模型如CLIP、Florence等推动了视觉-语言理解的突破，使得无需OCR即可直接处理文档图像，极大改善了理解复杂布局和多模态信息的能力。多向量检索策略通过将文档划分为多个patch嵌入，实现更细粒度的匹配，但存储成本随嵌入数量线性增长，成为实际部署的瓶颈。现有优化措施如聚类或池化，虽能减缓增长，但难以兼顾性能和效率，仍需创新方案突破。
Core Problem
多向量VDR模型在高精度检索中表现优异，但存储成本极高，尤其在大规模数据环境中难以部署。每个页面的patch嵌入数量以百计甚至千计，导致存储空间和计算资源消耗巨大，严重制约其应用推广。现有修剪方法多为固定比例或阈值，缺乏对不同类型文档的适应性，容易造成重要信息丢失或效率不足。如何在保证检索性能的同时，实现高效、动态的存储压缩，是亟待解决的核心问题。
Innovation
本文提出的创新点包括：1）利用Transformer模型最后一层全局Token的注意力分布，动态评估每个patch的重要性，避免固定阈值的局限；2）引入信息熵调节机制，根据文档内容的密度自适应调整修剪比例，增强鲁棒性；3）无需模型重训练，快速实现存储压缩，兼容多种模型架构。这些创新突破了传统静态修剪的限制，为多模态检索的存储优化提供了新思路。
Methodology
- �� 通过视觉-语言模型提取文档patch嵌入和全局Token注意力；
- �� 计算每个patch的注意力重要性分数，作为重要性指标；
- �� 利用统计特征（均值、标准差）自适应设定阈值，确保不同文档类型的关键特征得以保留；
- �� 根据阈值筛选patch，形成修剪集合，确保至少保留最重要的patch；
- �� 在检索阶段，仅用修剪后的patch集合进行相似度计算，提升效率；
- �� 理论基础结合信息瓶颈，确保信息最大化传递。
Experiments
在ViDoRe-V2和JinaVDR-Bench两个代表性数据集上，采用ColQwen2.5、ColNomic和Jina Embeddings V4模型，验证修剪效果。对比固定阈值、非自适应修剪和多种融合策略，使用nDCG@5指标。调节参数k，观察不同修剪比例对性能的影响。实验还涵盖多语种、多布局文档，确保方法的普适性。所有模型在NVIDIA A100 GPU上训练和测试，保证公平性。
Results
在多模型、多数据集环境中，DocPruner实现50-60%的patch压缩率，检索性能几乎无差异。例如，ColQwen2.5模型压缩54.1%，nDCG@5仅下降0.008（从0.5687到0.5608）；ColNomic模型压缩43.6%，性能保持在0.5946到0.5960之间。自适应阈值策略优于固定阈值和非调节方法，显著提升存储效率和鲁棒性。消融实验验证了全局Token注意力和信息熵调节的关键作用，确保信息最大保留。
Applications
该技术适用于大规模企业级文档检索系统，尤其在电子商务、数字图书馆和智能办公中，可显著降低存储成本，提升检索速度。未来结合端到端训练，将实现更智能的动态修剪，支持实时检索和多模态融合，推动行业智能化升级。
Limitations & Outlook
目前方法依赖Transformer模型的注意力机制，可能在非Transformer架构中效果有限。参数k的调节还需人工经验，可能影响不同场景的适应性。极端信息稀疏或密集的文档可能出现修剪过度或不足的问题。未来需研究更鲁棒的参数自适应机制和多模态特征融合策略，以应对更复杂的实际应用场景。
Plain Language Accessible to non-experts
想象你在整理一堆杂乱的文件夹，每个文件夹里有很多不同的文件。为了节省空间，你会挑出最重要的文件，把那些没那么重要的扔掉。这个过程就像DocPruner做的事，它会看每个文件夹里的每个文件，判断哪些内容最关键，然后只保留那些。这样，文件夹变得更小，但你仍然能找到你需要的内容。这个方法让存储变得更高效，也让检索变得更快，就像你用一个聪明的筛子，把重要的东西筛出来一样。
ELI14 Explained like you're 14
你知道当你整理书架时，会挑出最喜欢的书放在显眼的位置，把不常看的书放到一边？这就是在节省空间和找到想看的东西的技巧。DocPruner也是这样，它会看每个文档中的不同部分，判断哪些内容最重要，然后把那些内容留下来，其他的删掉。这样一来，存储空间变小了，找资料也更快了。这就像用一个聪明的剪刀，把不重要的部分剪掉，只留下最关键的内容，既节省空间，又不影响找到需要的资料。
Abstract
Visual Document Retrieval (VDR), the task of retrieving visually-rich document pages using queries that combine visual and textual cues, is crucial for numerous real-world applications. Recent state-of-the-art methods leverage Large Vision-Language Models (LVLMs) in a multi-vector paradigm, representing each document as patch-level embeddings to capture fine-grained details. While highly effective, this approach introduces a critical challenge: prohibitive storage overhead, as storing hundreds of vectors per page makes large-scale deployment costly and impractical. To address this, we introduce DocPruner, the first framework to employ adaptive patch-level embedding pruning for VDR to effectively reduce the storage overhead. DocPruner leverages the intra-document patch attention distribution to dynamically identify and discard redundant embeddings for each document. This adaptive mechanism enables a significant 50-60% reduction in storage for leading multi-vector VDR models with negligible degradation in document retrieval performance. Extensive experiments across more than ten representative datasets validate that DocPruner offers a robust, flexible, and effective solution for building storage-efficient, large-scale VDR systems.