Distilling a Small Utility-Based Passage Selector to Enhance Retrieval-Augmented Generation
Proposes utility-based passage selection via distillation, reducing large model costs while improving complex QA performance.
Key Findings
Methodology
The approach employs a teacher-student framework, distilling Qwen3-32B’s utility judgment into Qwen3-1.7B. It uses a sliding window strategy with pseudo-answer generation to enhance utility evaluation. Multi-task training enables the student to learn both pseudo-answering and utility assessment, significantly reducing inference costs. Experiments on MS MARCO’s 100k queries show successful distillation into RankQwen1.7B (relevance ranking) and UtilityQwen1.7B (utility selection), validated on NQ and HotpotQA datasets.
Key Results
- In HotpotQA, utility-based selection improves answer EM by 3.2% and F1 by 2.8%, outperforming relevance ranking. In NQ, differences are negligible, indicating better performance in complex scenarios.
- Dynamic filtering reduces computational costs by ~70%, achieving comparable or better results with only 30% of the resources.
- The distilled models generalize well across benchmarks like BEIR, especially in multi-hop reasoning tasks, accurately identifying useful passages.
Significance
This work addresses the high computational cost of large models in complex QA, offering a scalable, efficient solution. By transferring utility judgment capabilities into smaller models, it facilitates deployment in real-world applications, bridging the gap between performance and efficiency.
Technical Contribution
Introduces a sliding window-based distillation method combining pseudo-answer generation with utility assessment, enabling dynamic, task-adaptive filtering. This departs from traditional relevance ranking, providing a novel, efficient approach for large-scale passage selection.
Novelty
First to distill large-model utility judgment capabilities into lightweight models, demonstrating superior performance in multi-hop and complex QA, distinct from prior relevance-focused approaches.
Limitations
- Model accuracy drops in extremely ambiguous or novel questions, mainly due to pseudo-answer generation limitations.
- Training relies heavily on high-quality teacher outputs, incurring high initial costs.
- Cross-lingual and cross-domain robustness remains to be validated, requiring further dataset expansion.
Future Work
Future directions include multi-task joint training, reinforcement learning for adaptive filtering, and multi-lingual/multi-domain generalization, aiming to enhance robustness and broader applicability.
AI Executive Summary
This study introduces a novel utility-based passage selection framework aimed at overcoming the high computational costs associated with large language models in complex question answering. Traditional retrieval-augmented generation systems primarily rely on relevance ranking, which often falls short in complex scenarios where the usefulness of retrieved passages directly impacts answer quality. To address this, the authors design a sliding window strategy combined with pseudo-answer generation, enabling dynamic, query-adaptive filtering of candidate passages. The core innovation lies in distilling the utility judgment capability of a large teacher model, Qwen3-32B, into a compact student model, Qwen3-1.7B. This process involves multi-task training, where the student learns both to generate pseudo-answers and to assess passage utility, resulting in a lightweight model capable of efficient, high-quality passage filtering.
Experimental results on datasets like MS MARCO, NQ, and HotpotQA demonstrate that the utility-based selection significantly outperforms traditional relevance ranking, especially in complex multi-hop questions. The distilled model reduces inference costs by approximately 70%, while maintaining or improving answer accuracy, with improvements of over 3% in EM and F1 scores in challenging datasets. These findings show that dynamic, utility-driven filtering enables more effective use of limited computational resources, making large-scale, high-accuracy question answering systems more feasible for real-world deployment.
The broader impact of this work lies in its potential to democratize access to powerful AI systems by reducing hardware requirements and operational costs. It also opens avenues for further research into multi-task distillation, reinforcement learning-based adaptive filtering, and multi-modal integration. Despite these advances, challenges remain in extending robustness across diverse languages and domains, and in further improving pseudo-answer generation for ambiguous queries. Future work will focus on addressing these limitations, aiming to develop universally adaptable, resource-efficient intelligent systems.
Deep Analysis
Background
随着大规模预训练模型的兴起，知识检索与问答系统不断演进。早期方法多依赖关键词匹配和相关性排序，如BM25、BERT Rankers。近年来，深度学习模型如DPR、ANCE提升了检索效果，但在复杂问答中仍存在效率瓶颈。大模型如GPT-4、Qwen系列具备强大推理能力，但计算成本极高，限制了实际应用。知识蒸馏技术逐渐成为降低模型复杂度的关键手段，已有研究将大模型的排序能力迁移到轻量模型。本文在此基础上，关注于效用判断的迁移，旨在提升复杂问答中的筛选效率与效果。
Core Problem
传统检索系统多关注相关性排序，忽视了段落对答案的实际贡献，导致在复杂场景中筛选效果不足。大模型虽能准确判断段落效用，但推理成本极高，难以大规模应用。固定阈值筛选缺乏适应性，不能动态调整筛选数量，影响答案质量。如何在保证筛选效果的同时，降低推理成本，成为核心难题。需要设计一种高效、动态的筛选机制，兼顾准确性和可扩展性，满足不同复杂度问题的需求。
Innovation
本研究创新点包括：1）提出基于滑动窗口的动态效用筛选策略，结合伪答案生成，增强筛选鲁棒性；2）采用多任务蒸馏，将大模型的效用判断和伪答案生成能力迁移到小模型，实现成本与性能平衡；3）引入任务自适应筛选机制，避免固定阈值限制，提升模型在不同复杂度问题中的适应性。这些创新突破了传统相关性排序的局限，为复杂问答提供了更优解决方案。
Methodology
- �� 以Qwen3-32B为教师模型，利用其生成的伪答案和效用判断作为训练目标。• 设计从前向后滑动的窗口策略，逐步筛选候选段落，确保全局信息利用。• 采用多任务训练，结合伪答案生成与效用判断，增强模型任务适应性。• 利用MS MARCO的100k查询样本进行蒸馏，训练Qwen3-1.7B模型。• 通过动态筛选策略，生成高效筛选器，实现快速段落过滤。• 在多个数据集验证模型性能，包括NQ、HotpotQA。
Experiments
采用MS MARCO和BEIR多个公开数据集，评估筛选模型在相关性和效用筛选中的表现。对比BM25、BERT等传统方法，重点考察答案的EM、F1和检索召回率。设置不同窗口大小（20、30）和步长（10、15），验证动态筛选的效果。训练采用Adam优化器，学习率5e-6，训练三轮，确保模型收敛。多场景测试验证模型在复杂问答和多跳推理中的适应性和效率。
Results
在HotpotQA中，效用筛选模型提升答案EM 3.2%，F1 2.8%，优于相关性排序。NQ中两者差异不显著，验证效用筛选在复杂场景中的优势。筛选策略显著降低推理成本，减少70%的计算资源，保持甚至超越传统方法的性能。模型在多跳推理任务中表现出更强的段落识别能力，验证其在复杂问答中的实用性。
Applications
该技术适用于智能客服、学术搜索、医疗问答等场景，特别是在需要高精度和多信息融合的复杂问答任务中。通过模型压缩和动态筛选，降低部署门槛，提升系统响应速度，满足实际应用中的效率和效果需求。未来结合多模态信息，推动多源信息融合的智能问答系统发展。
Limitations & Outlook
模型在极端复杂或模糊问题中仍存在效用判断偏差，主要由于伪答案生成的局限性。蒸馏过程依赖大量高质量教师模型输出，训练成本较高。模型在多语言、多领域适应性不足，未来需扩展多样化数据集和多任务训练策略以提升泛化能力。
Plain Language Accessible to non-experts
想象你在厨房准备一顿大餐。每次你都要挑选食材，但试吃所有食材太费时间。于是，你请一个聪明的小助手帮你筛选，只挑那些新鲜又适合做这道菜的食材。这个助手学会你的偏好，能快速判断哪些食材最合适，省时又省力。就像论文里的模型，通过学习大厨（大模型）的判断，变得更快更聪明，帮你做出美味佳肴。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，你要找到很多宝藏，但宝藏藏得很深，找起来很难。你有个聪明的朋友，他能帮你判断哪些地方最可能藏有宝藏，但他帮你判断每个地方都要花很多时间。于是，你们设计了一个聪明的小机器人，让它学会像你的朋友一样判断宝藏的可能性，但只用很少时间筛选出最可能的地点。这样，你就能用更少时间找到更多宝藏，游戏变得更有趣啦！
Abstract
Retrieval-augmented generation (RAG) enhances large language models (LLMs) by incorporating retrieved information. Standard retrieval process prioritized relevance, focusing on topical alignment between queries and passages. In contrast, in RAG, the emphasis has shifted to utility, which considers the usefulness of passages for generating accurate answers. Despite empirical evidence showing the benefits of utility-based retrieval in RAG, the high computational cost of using LLMs for utility judgments limits the number of passages evaluated. This restriction is problematic for complex queries requiring extensive information. To address this, we propose a method to distill the utility judgment capabilities of LLMs into smaller, more efficient models. Our approach focuses on utility-based selection rather than ranking, enabling dynamic passage selection tailored to specific queries without the need for fixed thresholds. We train student models to learn pseudo-answer generation and utility judgments from teacher LLMs, using a sliding window method that dynamically selects useful passages. Our experiments demonstrate that utility-based selection provides a flexible and cost-effective solution for RAG, significantly reducing computational costs while improving answer quality. We present the distillation results using Qwen3-32B as the teacher model for both relevance ranking and utility-based selection, distilled into RankQwen1.7B and UtilityQwen1.7B. Our findings indicate that for complex questions, utility-based selection is more effective than relevance ranking in enhancing answer generation performance. We will release the relevance ranking and utility-based selection annotations for the MS MARCO dataset, supporting further research in this area.