Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Models
Promptriever is the first instruction-trained dense retriever, achieving state-of-the-art results with nearly 500k instruction-augmented pairs.
Key Findings
Methodology
This work introduces Promptriever, a bi-encoder model based on LLaMA-2 7B, trained on a large-scale dataset (~500k instances) augmented with natural language instructions and instruction negatives. The dataset was generated by prompting Llama-3-70B and GPT-4 to produce diverse instructions and relevant/non-relevant passage pairs, emphasizing instruction following and robustness. The training process involved combining standard MS MARCO data with instruction-enhanced data, optimizing the model to respond accurately to complex, detailed instructions while maintaining retrieval performance. The model leverages instruction-conditioned relevance scoring, enabling dynamic query-specific control.
Key Results
- On FollowIR, Promptriever achieved a 14.3 point increase in p-MRR, reaching SoTA levels, and improved nDCG by 3.1 points compared to baseline models.
- In InstructIR, robustness to lexical variations increased by 12.9, demonstrating enhanced adaptability to query phrasing. The model also showed a 1.4 point average boost on BEIR via prompt-based hyperparameter tuning.
- Compared to RepLLaMA, Promptriever maintained competitive standard retrieval scores (e.g., nDCG@10) while significantly improving instruction-following capabilities, illustrating the effectiveness of instruction augmentation.
Significance
This research addresses the critical limitation of static retrieval models by enabling dynamic, instruction-based control, bridging the gap between language understanding and information retrieval. It paves the way for personalized, flexible search systems that can adapt to user-specific instructions in real-time, greatly enhancing user experience and system versatility. The ability to follow detailed instructions and perform prompt-based hyperparameter tuning introduces new avenues for model deployment in real-world applications such as conversational AI, personalized search, and multi-task retrieval systems.
Technical Contribution
The paper presents a novel training paradigm combining large-scale instruction generation, instruction negative mining, and prompt tuning within a bi-encoder framework. This approach allows dense retrievers to respond to natural language instructions effectively, a capability previously limited to language models. The integration of instruction-conditioned relevance scoring and instruction negatives enhances the model’s sensitivity to nuanced user requirements, setting a new standard for controllable retrieval systems. The methodology also demonstrates how instruction-following can be systematically incorporated into dense retrieval training, opening new research directions.
Novelty
This is the first work to embed instruction-following capabilities directly into dense retrieval models, moving beyond static semantic similarity. The use of synthetic instruction data, instruction negatives, and prompt-based hyperparameter tuning distinguishes this approach from prior methods that treat retrieval as a fixed task. The model’s ability to adapt dynamically to complex, instance-level instructions marks a significant innovation in the field, aligning dense retrieval with the flexible, context-aware nature of language models.
Limitations
- The model’s performance heavily depends on the quality and diversity of instruction data, which requires substantial computational resources for generation and filtering. This may limit scalability in resource-constrained environments.
- Despite improvements, the model still struggles with extremely ambiguous or highly complex instructions, indicating room for further enhancement in understanding nuanced user requirements.
- The current architecture is based on a bi-encoder, which may limit the modeling of query-document interactions compared to cross-encoders. Future work should explore more sophisticated architectures to further boost performance.
Future Work
Future research will explore multi-modal instruction tuning, integrating visual and speech cues to expand the model’s capabilities. Developing more efficient instruction generation and negative sampling methods will reduce training costs. Additionally, combining self-supervised learning and few-shot adaptation could further improve performance in low-resource scenarios. Extending the framework to multilingual and cross-lingual retrieval tasks is also a promising direction, aiming to build universally adaptable instruction-following retrieval systems.
AI Executive Summary
Traditional information retrieval systems primarily rely on static semantic matching, which often fails to meet the demands of personalized and complex user instructions. This limitation hampers user experience and restricts the flexibility of retrieval models. Addressing this challenge, the present work introduces Promptriever, a novel dense retrieval model trained on an instruction-augmented dataset of nearly 500,000 instances. The core innovation lies in integrating natural language instructions directly into the training process, enabling the model to interpret and respond to detailed, complex queries dynamically.
The methodology involves generating diverse instructions using advanced language models like Llama-3-70B and GPT-4, coupled with instruction negative mining to teach the model to distinguish relevant from irrelevant passages under specific instructions. The training process combines standard MS MARCO data with this instruction-rich dataset, optimizing the model to follow instructions while maintaining high retrieval accuracy. The architecture is based on a bi-encoder framework, which ensures efficiency and scalability.
Experimental results demonstrate that Promptriever outperforms existing dense retrievers such as RepLLaMA, achieving a 14.3-point increase in p-MRR on FollowIR, and significantly improving robustness to query phrasing variations. Moreover, the model exhibits promptable hyperparameter tuning, leading to an average 1.4-point boost on the BEIR benchmark without retraining. These findings highlight the potential of instruction-conditioned retrieval, bridging the gap between language understanding and information retrieval.
The broader impact of this work lies in enabling more flexible, user-centric search systems capable of adapting to individual needs in real-time. It opens new avenues for personalized AI assistants, conversational retrieval, and multi-task learning. Despite these advances, challenges remain in reducing data generation costs and improving understanding of highly ambiguous instructions. Future directions include multi-modal instruction tuning, efficiency improvements, and extending the framework to multilingual contexts, promising a new era of intelligent, adaptable retrieval systems.
Deep Dive
Plain Language Accessible to non-experts
想象你在一家厨房里做饭，传统的厨师只知道按照菜谱一步步操作，不能随意变换食材或调料。现在，Promptriever就像一个聪明的厨师，不仅能按照指令做菜，还能根据你的不同要求调整食谱，比如“少盐”、“多辣”或“只用有机食材”。它通过学习大量不同的指令和食谱，变得越来越灵活，能理解各种复杂的指示，帮你做出符合你口味的菜肴。这就像你告诉厨师“我想吃辣一点的麻婆豆腐”，它能立刻调整做法，满足你的需求。这个模型的核心在于，它不仅能记住菜谱，还能理解你说的各种特殊要求，变得像个会变魔术的厨师一样聪明。
ELI14 Explained like you're 14
想象你在学校里，有个超级聪明的朋友，他不仅能帮你做作业，还能根据你的要求改变答案，比如“用简单的话说”或者“讲得更详细一点”。这个朋友就像Promptriever一样，它不仅能帮你找到信息，还能理解你说的特别要求。比如你问“哪种火山喷发我没见过？”，它能根据你的指示，帮你找到关于还没有被观察到的火山喷发类型的资料。它学会了很多不同的指令，知道怎么根据你的不同需求调整答案，就像你让朋友帮你做不同风格的作业一样。这样一来，信息变得更贴合你的需要，你也能更快找到想要的答案。
Glossary
Instruction-tuning (指令调优)
在模型训练中加入自然语言指令，使模型学会根据不同指令调整输出内容。通过在训练数据中加入多样化指令，增强模型的指令理解能力。
本文中，指令调优用于提升模型对复杂指令的响应能力。
Bi-encoder (双编码器)
一种神经网络架构，将查询和文档分别编码成向量，通过相似度计算实现快速检索。适合大规模场景。
Promptriever采用基于LLaMA-2的双编码器架构。
MS MARCO
微软发布的大规模网页检索数据集，包含查询、相关文档和负样本，用于训练和评估检索模型。
本研究使用MS MARCO作为基础训练数据。
p-MRR (偏重指令遵循的平均排名指标)
衡量模型在指令遵循任务中排名性能的指标，范围-100到100，数值越高越好。
评估Promptriever在FollowIR和InstructIR上的指令遵循效果。
Open Questions Unanswered questions from this research
- 1 如何进一步降低指令生成和筛选成本，提升模型在极端复杂指令下的理解能力，是未来研究重点。
- 2 多模态指令（结合图像、语音等）下的表现仍未充分探索，未来需结合多模态信息提升理解能力。
Applications
Immediate Applications
个性化搜索引擎
用户通过自然语言指令调节搜索偏好，实现更精准的内容推荐和信息检索。需要高质量指令数据和实时调控机制。
智能问答系统
结合Promptriever，提供基于复杂指令的答案，提升用户体验，适用于客服、教育等场景。
Long-term Vision
多模态智能交互
融合视觉、语音等多模态信息，打造更智能、更自然的人机交互系统，未来可实现全场景自主调控。
Abstract
Instruction-tuned language models (LM) are able to respond to imperative commands, providing a more natural user interface compared to their base counterparts. In this work, we present Promptriever, the first retrieval model able to be prompted like an LM. To train Promptriever, we curate and release a new instance-level instruction training set from MS MARCO, spanning nearly 500k instances. Promptriever not only achieves strong performance on standard retrieval tasks, but also follows instructions. We observe: (1) large gains (reaching SoTA) on following detailed relevance instructions (+14.3 p-MRR / +3.1 nDCG on FollowIR), (2) significantly increased robustness to lexical choices/phrasing in the query+instruction (+12.9 Robustness@10 on InstructIR), and (3) the ability to perform hyperparameter search via prompting to reliably improve retrieval performance (+1.4 average increase on BEIR). Promptriever demonstrates that retrieval models can be controlled with prompts on a per-query basis, setting the stage for future work aligning LM prompting techniques with information retrieval.