Learning Passage Impacts for Inverted Indexes
DeepImpact leverages semantic impact scores with BERT and DocT5Query to improve first-stage retrieval by 17%, enabling faster and more accurate search.
Key Findings
Methodology
DeepImpact combines document expansion via DocT5Query with a BERT-based neural model to directly estimate semantic impact scores for each token. The model is trained end-to-end using triplet sampling from MS MARCO, optimizing the sum of impact scores to maximize the relevance gap. Impact scores are quantized for storage efficiency in inverted indexes. During retrieval, scores are summed for query terms, enabling efficient scoring. The approach improves impact modeling over traditional term frequency methods, effectively addressing vocabulary mismatch and boosting retrieval effectiveness by up to 17%. In re-ranking scenarios, it matches state-of-the-art accuracy with 5.1Γ speedup.
Key Results
- On MS MARCO passage ranking, DeepImpact achieved an MRR@10 of 0.326, outperforming BM25 (0.188), DeepCT (0.244), and DocT5Query (0.278), with a 17% improvement in effectiveness metrics.
- Across TREC 2019/2020 datasets, DeepImpact consistently outperformed baseline methods in NDCG@10 and MAP, with significant gains in re-ranking tasks, and reduced latency by 5.1Γ when combined with ColBERT.
- The model maintains high recall at small candidate set sizes, especially at top 10 or 20, facilitating efficient downstream re-ranking and reducing computational costs.
Significance
This work introduces a novel way to incorporate deep semantic understanding into traditional inverted indexes, effectively bridging the gap between high-accuracy neural models and scalable retrieval systems. By directly estimating token impact scores, it addresses the core challenge of vocabulary mismatch and semantic gap, making neural retrieval practical for large-scale deployment. Its efficiency and effectiveness open new avenues for search engines, question answering, and personalized recommendation systems, significantly enhancing user experience and system performance.
Technical Contribution
The key technical innovation lies in training a BERT-based model to produce impact scores for tokens, optimized jointly across all query terms, and storing these scores in a quantized form within inverted indexes. This approach enables fast scoring during query processing while capturing rich semantic interactions. Unlike prior methods that rely solely on term frequency or independent impact estimation, DeepImpact models the combined influence of multiple query terms, providing a more holistic relevance measure. The integration of document expansion with impact score learning further enhances retrieval robustness.
Novelty
DeepImpact is the first to directly learn and store semantic impact scores for tokens in an inverted index, combining document expansion via DocT5Query with a neural impact scoring model trained end-to-end. Unlike DeepCT, which performs per-token regression without considering impact interactions, DeepImpact jointly optimizes impacts across all query terms, effectively addressing vocabulary mismatch and semantic relevance. This represents a significant step forward in neural IR, blending deep semantic understanding with scalable indexing.
Limitations
- The impact scoring relies on large-scale annotated triplet data and may not generalize well to unseen domains or languages without retraining. Quantization may introduce minor information loss affecting retrieval precision.
- Impact score distribution can be skewed, reducing the efficiency of algorithms like MaxScore, especially with non-uniform score ranges. Further optimization of score distribution is needed.
- The current approach is primarily validated on English datasets; multilingual and cross-domain adaptation remain open challenges, requiring additional research.
Future Work
Future efforts will focus on refining impact score distribution to optimize retrieval algorithms, exploring multi-modal document expansion techniques, and extending the approach to multilingual settings. Additionally, integrating more sophisticated impact modeling, such as context-aware impact interactions, and reducing training data requirements are promising directions to enhance scalability and robustness.
AI Executive Summary
In the era of information explosion, search engines face the challenge of retrieving relevant documents swiftly and accurately from vast datasets. Traditional inverted indexes, based on keyword matching and term frequency, struggle to capture the nuanced semantic relationships between queries and documents, leading to the persistent problem of vocabulary mismatch. Deep learning approaches, especially models like BERT, have demonstrated remarkable improvements in understanding language semantics but come with prohibitive computational costs, hindering their deployment at scale.
To bridge this gap, the paper introduces DeepImpact, a novel framework that combines document expansion via DocT5Query with a neural impact scoring model based on BERT. The core idea is to directly estimate the semantic importance of each token in a document, producing a single impact score per token that can be stored efficiently in an inverted index. This impact score reflects how much a token contributes to the relevance of a document for potential queries. During training, the model is optimized end-to-end using triplet data from MS MARCO, where the sum of impact scores across query terms is maximized to distinguish relevant from non-relevant documents.
The impact scores are quantized into compact integers, enabling fast retrieval during query processing by summing the scores of matching terms. Extensive experiments on MS MARCO and TREC datasets show that DeepImpact outperforms traditional methods like BM25 and DeepCT, achieving a 17% improvement in effectiveness metrics. When integrated into a re-ranking pipeline with models like ColBERT, it maintains high accuracy while reducing latency by over five times. This approach effectively addresses the vocabulary mismatch problem, enhances the expressiveness of first-stage retrieval, and offers a scalable, efficient solution for large-scale neural information retrieval.
Overall, DeepImpact represents a significant advancement in neural IR, blending deep semantic understanding with traditional indexing techniques. Its ability to produce high-quality, impact-based document representations paves the way for faster, more accurate search systems, with broad implications for search engines, question answering, and personalized recommendation platforms. Future work aims to refine impact score distributions, extend multilingual capabilities, and incorporate multi-modal data, further broadening its applicability and robustness.
Deep Dive
Abstract
Neural information retrieval systems typically use a cascading pipeline, in which a first-stage model retrieves a candidate set of documents and one or more subsequent stages re-rank this set using contextualized language models such as BERT. In this paper, we propose DeepImpact, a new document term-weighting scheme suitable for efficient retrieval using a standard inverted index. Compared to existing methods, DeepImpact improves impact-score modeling and tackles the vocabulary-mismatch problem. In particular, DeepImpact leverages DocT5Query to enrich the document collection and, using a contextualized language model, directly estimates the semantic importance of tokens in a document, producing a single-value representation for each token in each document. Our experiments show that DeepImpact significantly outperforms prior first-stage retrieval approaches by up to 17% on effectiveness metrics w.r.t. DocT5Query, and, when deployed in a re-ranking scenario, can reach the same effectiveness of state-of-the-art approaches with up to 5.1x speedup in efficiency.