Incorporating Token Importance in Multi-Vector Retrieval

Recent progress in natural language understanding, driven by models like BERT, has transformed information retrieval. Dense retrieval methods encode queries and documents into high-dimensional vectors, enabling semantic matching. Early models like DPR used simple vector averaging, but lacked fine-grained semantic sensitivity. ColBERT introduced token-level multi-vector representations, allowing late interaction scoring via distance functions, which improved expressiveness and efficiency. Subsequent works optimized indexing and training strategies, such as ColBERTv2, to handle billion-scale corpora. Despite these advances, most models treat all tokens equally, ignoring their varying importance in relevance estimation, which limits their ability to capture nuanced semantic signals, especially in out-of-domain or resource-scarce scenarios.

The core challenge addressed is the uniform weighting of tokens in multi-vector retrieval models, which neglects the fact that some tokens contribute more significantly to relevance than others. Traditional models like BM25 leverage term frequency and inverse document frequency (IDF) to weigh tokens, but dense models lack such explicit importance modeling. This leads to suboptimal matching, especially for long or ambiguous texts. Additionally, existing models often require extensive labeled data for fine-tuning, which is costly and impractical in many real-world applications. Therefore, developing a lightweight, effective method to incorporate token importance into dense retrieval remains a critical problem.

The main innovation is the integration of token importance weights into the late interaction distance function, specifically the Chamfer distance, creating Weighted Chamfer. This approach allows the model to emphasize more relevant tokens during similarity computation. The importance weights can be derived statically using IDF, requiring no additional training, or learned via contrastive loss with limited relevance data. This method maintains the original encoder parameters, avoiding retraining, and introduces minimal computational overhead. Theoretically, the authors prove bounds on sample complexity for weight recovery and generalization, providing a solid foundation for the approach. The simplicity and effectiveness of this design make it a practical enhancement for existing dense retrieval systems.

• �� Encode queries and documents into token-level vectors using fixed BERT-based encoders.
• �� Define Weighted Chamfer distance by scaling each query token distance with a learnable weight.
• �� For zero-shot, compute static weights via IDF statistics from the document corpus.
• �� For few-shot, optimize token weights using contrastive loss to minimize distances to relevant documents and maximize to negatives.
• �� Use iterative negative sampling to focus on hard negatives, improving weight learning.
• �� Keep encoder parameters fixed; only train token weights, simplifying optimization.
• �� Theoretically analyze sample complexity and generalization bounds for weight recovery and supervised learning.

Experiments conducted on BEIR benchmark, using BM25 retrieval + ColBERTv2 re-ranking. Zero-shot weights derived from IDF, evaluated across 13 datasets with metrics like Recall@10, MRR@10, nDCG@10. Fine-tuning performed with limited relevance data, showing consistent improvements. Hyperparameters such as negative set sizes and learning rates tuned via validation. Results demonstrate that static IDF weights improve Recall@10 by 1.28%, while supervised fine-tuning boosts it to 3.66%. The approach maintains low inference latency, scales well across datasets, and outperforms baseline models, especially in out-of-domain scenarios.

IDF-based static weights improve Recall@10 by 1.28% on average, with maximum gains over 3.16%. Fine-tuning with limited relevance data further enhances performance to 3.66%. The method consistently outperforms original ColBERTv2 in multiple metrics, validating the importance of token weighting. Theoretical analysis confirms that the approach requires fewer samples for effective weight recovery and generalizes well across datasets. Results highlight the robustness of token importance weighting in improving semantic matching, especially in resource-constrained or cross-domain settings.

Applicable in search engines, question answering, and knowledge bases, especially where labeled relevance data is scarce. The lightweight nature allows easy integration into existing pipelines, boosting accuracy without retraining large models. It is particularly useful for low-resource languages or domains with limited annotations. The method can also serve as a foundation for dynamic, context-aware importance weighting, further enhancing retrieval precision in real-time applications.

The static importance weights rely on corpus statistics and may not adapt well to domain shifts or emerging vocabularies. Supervised fine-tuning depends on relevance annotations, which are costly. The approach does not incorporate dynamic context, potentially limiting performance in complex semantic scenarios. In texts with dense polysemy or ambiguous tokens, importance estimates may be inaccurate, affecting retrieval quality. Future work should explore context-sensitive importance modeling and unsupervised learning strategies.