Aligning Dense Retrievers with LLM Utility via DistillationAligning Dense Retrievers with LLM Utility via Distillation

In the field of information retrieval, dense vector retrieval has become the cornerstone of Retrieval-Augmented Generation (RAG) systems. These systems map queries and candidates into a shared representation space, leveraging efficient Approximate Nearest Neighbor (ANN) search to handle large-scale datasets with minimal latency. However, as technology advances, this paradigm is increasingly criticized for its reliance on semantic similarity as a proxy for generative utility. Research shows that passages with high semantic similarity (topical overlap) often fail to provide answer-critical information and can even introduce semantic distractors that mislead the generator, especially in long-context settings where incorrect but similar passages increase decoding uncertainty. To bridge this gap, current state-of-the-art approaches shift towards utility-based retrieval, where relevance is defined by how effectively a document helps a Large Language Model (LLM) produce a correct response. In practice, this is often measured via perplexity reduction: a document is considered useful if its presence as context makes the ground-truth answer more predictable.

Despite the conceptual soundness of utility-based approaches, they face significant practical challenges. Relying on LLMs for query generation or post-hoc re-ranking is computationally prohibitive for large-scale deployment. Furthermore, utility signals derived from perplexity are notoriously noisy and stochastic, sensitive to token-level variations and decoding dynamics that make them difficult to use as stable training targets. This necessitates complex, multi-stage architectures that improve performance at the cost of extreme inference latency and high computational overhead. Therefore, the challenge lies in enhancing retrieval's generative utility without increasing computational burden.

The core innovation of the UAE framework lies in distilling generative utility directly into the bi-encoder's embedding space, thus avoiding expensive test-time LLM inference. Specifically:

• �� UAE formulates retrieval as a distribution matching problem, using a Utility-Modulated InfoNCE objective to train the bi-encoder to mimic a utility distribution derived from perplexity reduction. This approach injects graded utility signals directly into the embedding space without requiring test-time LLM inference.

• �� UAE stabilizes noisy utility signals by distilling them into a parameterized reward model during training, aligning the dense retriever with this model through supervised distribution matching. This method maintains standard ANN compatibility, providing a practical and scalable solution for utility-driven RAG systems.

• �� UAE significantly improves retrieval performance on the QASPER benchmark while being over 180x faster than efficient LLM re-ranking methods, demonstrating that aligning retrieval with generative utility yields reliable contexts at scale.

The implementation of UAE involves several key steps:

• �� Parameterized Utility Approximation: Estimate the utility of context documents via perplexity and distill them into a parameterized reward model to stabilize noisy utility signals.

• �� Reward-Guided Embeddings Training: Use the reward model as an offline teacher to distill utility preferences into a dense bi-encoder, preserving ANN search efficiency.

• �� Distribution Matching Objective: Define a teacher distribution and a student distribution, optimizing the retriever by minimizing the KL divergence between the two distributions, reshaping the embedding space to reflect the generator's preferences.

• �� Utility-Aware Hard Negative Mining: Adopt the Noise Contrastive Estimation (NCE) paradigm to approximate the global distribution using a combination of gold contexts and informative negative samples, ensuring the retriever focuses on resolving semantic distractors.

The experimental design includes evaluation on two distinct RAG benchmarks: QASPER (long-doc scientific QA) and NewsQA (short-doc news extraction). We employ a hard-negative setting where the candidate pool (50) for each query is constructed via dense retrieval (BGE-Base) and reward model utility. This setting rigorously tests the model's ability to prioritize true generative utility over high-similarity non-answers. The generation protocol uses Llama-3-8B-Instruct as the fixed generator with greedy decoding (temperature=0) for reproducibility. Dataset-specific system prompts align the generator's output with the ground-truth format: extractive phrases for NewsQA and evidence-based summaries for QASPER. Performance is quantified using Token F1 and ROUGE-L to assess informational accuracy and structural fluency.

On the QASPER benchmark, UAE improves retrieval Recall@1 by 30.59%, MAP by 30.16%, and Token F1 by 17.3% over the strong semantic baseline BGE-Base. UAE is over 180x faster than efficient LLM re-ranking methods while preserving competitive performance, demonstrating that aligning retrieval with generative utility yields reliable contexts at scale. In experiments, UAE achieves a Recall@1 of 54.90 on the NewsQA dataset, surpassing the computationally expensive RankGPT (49.68), indicating that embeddings aligned with generative utility can achieve reranker-level precision in a single retrieval step. This advantage extends to ExpUtil@1 (average utility of the top-1 context). On NewsQA, UAE (5.818) surpasses both BGE-Base (4.738) and even the computation-heavy RankGPT (4.968), confirming that UAE prioritizes contexts maximally conducive to generation rather than mere semantic relevance.

Direct application scenarios for UAE include:

• �� In large-scale information retrieval systems, UAE can serve as an efficient first-stage retriever, providing high-quality candidate contexts for subsequent processing.

• �� In real-time applications requiring quick responses, UAE's low latency makes it ideal, especially in scenarios where user experience is critical.

• �� In complex tasks requiring long-context processing, UAE can improve generation quality by reducing semantic distractors, applicable in scientific literature analysis and long-document question answering.

UAE may struggle with very long contexts as they can introduce more semantic distractors, affecting generation quality. Additionally, the performance of UAE may be limited by the quality of the pre-trained generative model it relies on. In certain domains or tasks, UAE may require additional fine-tuning to achieve optimal performance. Future research directions include exploring UAE's application across different domains and tasks, further improving its performance with long contexts, and investigating ways to enhance UAE's generative quality without compromising efficiency. Additionally, exploring how to integrate other advanced retrieval techniques to further enhance UAE's performance and applicability is a promising direction.