Bottleneck Tokens for Unified Multimodal Retrieval
Introduces Bottleneck Tokens (BToks) to address structural issues in multimodal retrieval, achieving a score of 59.0 on MMEB-V2.
Key Findings
Methodology
This study proposes a novel multimodal retrieval method using Bottleneck Tokens (BToks) as an explicit pooling mechanism, coupled with a generative information condensation training objective. BToks are learnable tokens used to aggregate and compress information in multimodal inputs. Generative information condensation provides token-level supervision through a generative objective and structural attention constraint.
Key Results
- On the MMEB-V2 benchmark, the BToks method achieved an overall score of 59.0 among 2B-scale models, improving by 3.6 points over VLM2Vec-V2, with a notable 12.6-point gain in video QA tasks.
- In image tasks, BToks improved image QA and grounding tasks by 3.6 and 2.7 points, respectively, demonstrating advantages in semantically complex tasks.
- Ablation studies showed performance drops when removing bottleneck tokens or the generative objective, validating the effectiveness of each component.
Significance
This study is significant in the field of multimodal retrieval, addressing structural deficiencies in existing methods that rely on implicit pooling. By introducing explicit bottleneck tokens and generative information condensation, the model performs better in complex semantic tasks, advancing multimodal retrieval technology.
Technical Contribution
Technical contributions include: 1) Introducing bottleneck tokens as a dedicated pooling mechanism, addressing the structural issues of implicit pooling; 2) Proposing a generative information condensation training mechanism that provides token-level supervision for multimodal information compression; 3) Significantly improving model performance in complex tasks without increasing inference complexity.
Novelty
This study is the first to combine explicit bottleneck tokens with generative information condensation, providing a fixed-capacity information bottleneck for aggregating and compressing multimodal information. This approach fundamentally differs from existing implicit pooling methods.
Limitations
- In cases of long sequence inputs, although bottleneck tokens provide a fixed-capacity pooling mechanism, there may still be issues with information loss.
- The training process for generative information condensation may require extended computation time.
Future Work
Future research directions include exploring more efficient generative information condensation mechanisms, optimizing computational resource usage, and applying this method to larger datasets and more diverse tasks.
AI Executive Summary
Multimodal retrieval is a key technology for encoding text and visual information into a shared embedding space. However, existing methods rely on implicit pooling, leading to insufficient information aggregation. This paper proposes a new method using Bottleneck Tokens (BToks) as an explicit pooling mechanism, coupled with a generative information condensation training objective. BToks are learnable tokens used to aggregate and compress information in multimodal inputs. Generative information condensation provides token-level supervision through a generative objective and structural attention constraint.
On the MMEB-V2 benchmark, this method achieved an overall score of 59.0 among 2B-scale models, improving by 3.6 points over VLM2Vec-V2, with a notable 12.6-point gain in video QA tasks. This indicates that BToks can significantly improve model performance in complex tasks without increasing inference complexity.
Nevertheless, this method may face issues with information loss in long sequence inputs. Additionally, the training process for generative information condensation may require extended computation time. Future research directions include exploring more efficient generative information condensation mechanisms and optimizing computational resource usage.
Deep Analysis
Background
Multimodal retrieval technology has gained significant attention in recent years, with its core being the encoding of text and visual information into a shared embedding space. Traditional methods often rely on implicit pooling mechanisms, such as using the hidden state of the <EOS> token as a global representation. However, this approach performs poorly in complex semantic tasks because the <EOS> token was not designed for information aggregation.
Core Problem
Existing multimodal retrieval methods have structural deficiencies in information aggregation, especially in tasks requiring precise semantic compression. Implicit pooling methods fail to provide token-level guidance for information compression, making it difficult for models to effectively capture query intent in complex tasks.
Innovation
The core innovation of this paper is the introduction of Bottleneck Tokens (BToks) as an explicit pooling mechanism, addressing the structural issues of implicit pooling. BToks are learnable tokens used to aggregate and compress multimodal information. Additionally, the generative information condensation training mechanism provides token-level supervision through a generative objective and structural attention constraint.
Methodology
- �� Introduce Bottleneck Tokens (BToks) as a fixed-capacity explicit pooling mechanism.
- �� Design a generative information condensation training objective, providing token-level supervision through a generative objective and structural attention constraint.
- �� During inference, only process inputs and BToks, maintaining inference complexity.
Experiments
Experiments were conducted on the MMEB-V2 benchmark, covering 78 datasets and 9 meta-tasks. A 2B-scale model was used for comparative experiments, evaluating the performance of BToks in different tasks. Ablation studies validated the effectiveness of each component.
Results
BToks achieved an overall score of 59.0 on MMEB-V2, improving by 3.6 points over VLM2Vec-V2, with a notable 12.6-point gain in video QA tasks. Ablation studies showed performance drops when removing bottleneck tokens or the generative objective.
Applications
This method can be directly applied to multimodal retrieval tasks, such as image QA and video retrieval. Its explicit pooling mechanism is particularly effective for complex semantic tasks, with broad industry application potential.
Limitations & Outlook
Although BToks provide an explicit pooling mechanism, there may be issues with information loss in long sequence inputs. Additionally, the training process for generative information condensation may require extended computation time.
Plain Language Accessible to non-experts
Imagine a factory responsible for processing various raw materials into products. Traditional methods are like putting all materials in a big box, hoping the final product represents all the information. Bottleneck Tokens (BToks) are like specialized workshops, each token handling specific information, ensuring the final product accurately reflects the raw materials' characteristics. Generative information condensation is like a quality control process, ensuring each workshop effectively processes information, preventing information loss.
ELI14 Explained like you're 14
Imagine you're playing a game that requires handling both text and pictures. Traditional methods are like throwing all the information to one brain, hoping it remembers everything. But that's hard! Bottleneck Tokens (BToks) are like little helpers in the game, each handling a part of the information, ensuring you quickly find the answer. Generative information condensation is like a smart hint system, helping these helpers work better.
Glossary
Bottleneck Tokens
A set of learnable tokens used to aggregate and compress multimodal information.
Used as an explicit pooling mechanism in the paper.
Generative Information Condensation
A training mechanism combining a generative objective and structural attention constraint.
Provides token-level supervision in the paper.
Implicit Pooling
Using the hidden state of the <EOS> token as a global representation.
A flaw in traditional multimodal retrieval methods.
Contrastive Fine-tuning
A method of optimizing embedding models through contrastive loss.
Used in the training process of the model.
MMEB-V2
A large-scale multimodal embedding benchmark covering 78 datasets and 9 meta-tasks.
Used to evaluate model performance.
Open Questions Unanswered questions from this research
- 1 How to maintain information integrity in long sequence inputs? Current methods may lead to information loss, requiring more efficient compression mechanisms.
- 2 How to improve the computational efficiency of generative information condensation? The current method may require extended training time.
Applications
Immediate Applications
Multimodal Retrieval
Can be used for image QA and video retrieval, enhancing performance in complex tasks.
Long-term Vision
Intelligent Search Engines
Improved multimodal retrieval technology enhances the semantic understanding capabilities of search engines.
Abstract
Adapting decoder-only multimodal large language models (MLLMs) for unified multimodal retrieval faces two structural gaps. First, existing methods rely on implicit pooling, which overloads the hidden state of a standard vocabulary token (e.g., <EOS>) as the sequence-level representation, a mechanism never designed for information aggregation. Second, contrastive fine-tuning specifies what the embedding should match but provides no token-level guidance on how information should be compressed into it. We address both gaps with two complementary components. Architecturally, we introduce Bottleneck Tokens (BToks), a small set of learnable tokens that serve as a fixed-capacity explicit pooling mechanism. For training, we propose Generative Information Condensation: a next-token prediction objective coupled with a Condensation Mask that severs the direct attention path from target tokens to query tokens. All predictive signals are thereby forced through the BToks, converting the generative loss into dense, token-level supervision for semantic compression. At inference time, only the input and BToks are processed in a single forward pass with negligible overhead over conventional last-token pooling. On MMEB-V2 (78 datasets, 3 modalities, 9 meta-tasks), our approach achieves state-of-the-art among 2B-scale methods under comparable data conditions, attaining an Overall score of 59.0 (+3.6 over VLM2Vec-V2) with substantial gains on semantically demanding tasks (e.g., +12.6 on Video-QA).