Pre-computed memory or on-the-fly encoding? A hybrid approach to retrieval augmentation makes the most of your compute
LUMEN combines pre-computed and on-the-fly encoding, improving QA performance with reduced compute, especially at larger scales.
Key Findings
Methodology
LUMEN employs a hybrid approach, dividing the encoder into a frozen MemoryEncoder and a fine-tuned LiveEncoder. Precomputed passage representations are stored, while a small portion of the encoder dynamically updates these representations conditioned on the input question via a question encoder. The model is based on T5, with the encoder split into parts, enabling most of the retrieval representation to be precomputed, reducing inference FLOPs. Experiments on Natural Questions and TriviaQA demonstrate that with only a fraction of live layers, LUMEN surpasses pure memory models and approaches FiD performance at a fraction of the cost, with benefits increasing as model size grows.
Key Results
- In T5-XXL, LUMEN achieves an Exact Match of 57.1%, close to FiD, using only one-third of the dynamic layers, outperforming MemoryFiD by about 10%. Across model scales (Base, Large, XL, XXL), LUMEN consistently outperforms pure memory approaches at the same FLOPs, with performance gains exceeding 20%. Transfer learning from Natural Questions further enhances adaptability, especially on small datasets like WebQuestions. The results show that small dynamic proportions (around 1/3) suffice for near-FiD performance, with larger models requiring fewer dynamic layers, thus improving efficiency.
- Transfer learning from related tasks significantly boosts performance, especially for smaller dynamic proportions, reducing the performance gap with FiD. Fine-tuning the question encoder and transferring pre-trained memory and live encoders from Natural Questions help adapt the model to new tasks with limited data, demonstrating strong generalization.
- Performance scales favorably with model size; larger models need fewer dynamic layers to match FiD, and the FLOPs required for LUMEN are substantially lower than FiD at comparable performance levels. The architecture’s flexibility allows for efficient scaling, making it suitable for large-scale knowledge-intensive applications.
Significance
This work addresses the fundamental challenge of balancing computational cost and model performance in retrieval-augmented language models. By integrating precomputed representations with minimal dynamic updates conditioned on input, LUMEN offers a scalable solution for knowledge-intensive tasks, enabling deployment in real-world scenarios with limited resources. Its architecture paves the way for more efficient large-scale models capable of leveraging vast knowledge bases without prohibitive costs, impacting both academia and industry by making sophisticated question-answering systems more accessible and practical.
Technical Contribution
LUMEN introduces a novel hybrid encoder structure that partitions the T5 encoder into static and dynamic components, leveraging question-conditioned updates to precomputed passage representations. This design reduces inference FLOPs significantly while maintaining high accuracy. The architecture also incorporates transfer learning strategies to improve adaptability across tasks. The approach offers a new paradigm for balancing cost and performance in retrieval-augmented models, expanding the engineering possibilities for large-scale knowledge systems.
Novelty
This is the first work to systematically combine precomputed dense representations with minimal on-the-fly updates conditioned on input questions, achieving near-FiD performance at a fraction of the computational cost. Unlike prior methods that rely solely on static memory or full dynamic encoding, LUMEN’s hybrid approach dynamically adjusts representations with limited layers, representing a significant innovation in retrieval-augmented modeling.
Limitations
- Despite reduced FLOPs, the storage requirements for precomputed representations remain large, especially for extensive corpora, posing practical challenges in deployment. The model's performance depends on the quality of precomputed representations and the tuning of the dynamic layer proportion, which may vary across tasks. Additionally, in scenarios with rapidly changing knowledge bases, frequent re-computation of memory representations could offset efficiency gains. Future work should focus on compression techniques and adaptive dynamic layer selection to address these issues.
Future Work
Future directions include optimizing storage and retrieval mechanisms, such as compressed representations or hierarchical indexing, to handle larger knowledge bases efficiently. Exploring adaptive dynamic layer proportions based on input complexity could further improve efficiency. Extending the architecture to multi-modal data and real-time knowledge updates will broaden its applicability. Additionally, integrating reinforcement learning for dynamic layer tuning and representation refinement could enhance robustness and generalization across diverse tasks.
AI Executive Summary
The rapid advancement of large-scale pre-trained language models has revolutionized natural language understanding, especially in knowledge-intensive tasks like question answering. However, the high computational cost associated with models such as Fusion-in-Decoder (FiD) limits their practical deployment, particularly when processing vast external knowledge bases. To address this, the paper introduces LUMEN, a hybrid retrieval architecture that intelligently combines precomputed dense representations with minimal on-the-fly updates.
LUMEN’s core innovation lies in dividing the encoder into a frozen MemoryEncoder and a fine-tuned LiveEncoder. The MemoryEncoder precomputes passage representations offline, stored for quick retrieval. During inference, a question encoder generates a question representation, which, combined with the stored passage representations, undergoes limited dynamic updating via the LiveEncoder. This process allows the model to adapt representations conditioned on the input, significantly reducing the number of layers that need to be dynamically processed.
Experimental results on datasets like Natural Questions and TriviaQA demonstrate that LUMEN achieves near-FiD performance with only a third of the dynamic layers, translating into substantial computational savings—up to 20% fewer FLOPs at comparable accuracy. Larger models benefit even more, with the performance gap narrowing further. Transfer learning experiments show that pretraining on related tasks enhances adaptability, especially in low-data scenarios.
This architecture offers a compelling solution for deploying knowledge-intensive models efficiently, balancing the trade-off between cost and accuracy. By enabling models to dynamically update precomputed representations with minimal overhead, LUMEN paves the way for scalable, resource-efficient AI systems capable of leveraging extensive knowledge bases in real-time applications. Its flexible design and promising results mark a significant step toward practical, high-performance retrieval-augmented language models.
Deep Analysis
Background
Recent developments in NLP have seen the rise of large pre-trained models like T5, GPT, and BERT,推动了知识密集型任务的性能提升。检索增强模型如FiD通过引入外部知识库显著改善答案准确率，但其高昂的编码成本成为瓶颈。为降低推理成本，研究者尝试将知识库预先编码为密集向量存储（Memory），以实现快速检索，但这牺牲了表示的动态适应性，导致性能下降。近年来，结合预存表示与动态更新的混合架构逐渐成为研究热点，旨在兼顾效率与效果。本文在此基础上提出LUMEN，结合预存表示与少量动态层，优化知识问答的效率与性能，填补了现有方案的空白。
Core Problem
现有检索增强模型在高性能和低成本之间难以兼得。FiD模型虽效果优异，但每次推理都需编码大量检索内容，计算成本高昂。预存表示方案虽降低了实时编码负担，但因表示不受输入条件影响，导致性能不足，尤其在知识更新频繁或任务变化时表现不佳。如何设计一种架构，既能保持高性能，又能降低成本，成为亟待解决的问题。特别是在大规模知识库和多任务场景中，如何实现表示的高效存储与动态调节，是当前的技术难点。
Innovation
本文提出LUMEN架构，创新点包括：1）将编码器划分为静态的MemoryEncoder和动态的LiveEncoder，减少推理时的计算量；2）引入问句条件化机制，使存储表示能根据输入动态调整；3）结合迁移学习，提升模型在不同任务中的适应性。模型在预存表示基础上，通过少量动态层实现条件化更新，极大降低推理FLOPs，同时保持高准确率。这一设计突破了纯Memory和FiD的局限，为大模型知识增强提供了新思路。
Methodology
- �� 采用预训练T5模型，将编码器划分为MemoryEncoder（前部分层）和LiveEncoder（后部分层）；
- �� 预先对知识库中的段落进行编码，存储为静态表示；
- �� 输入问句经过问句编码器生成表示，结合存储表示，进行少量动态更新；
- �� 仅在α比例的层进行条件化编码，显著降低推理FLOPs；
- �� 训练过程中，微调问句编码器和动态层，保持存储表示不变；
- �� 推理时，结合问句表示与存储表示，生成最终输出。
Experiments
在Natural Questions和TriviaQA数据集上，比较LUMEN、FiD和MemoryFiD的性能，模型规模涵盖Base、Large、XL、XXL。训练采用T5X框架，优化器为Adafactor，批次64，学习率0.0001。评估指标为Exact Match（EM）和FLOPs，分析不同动态层比例（α）对性能的影响。还进行了迁移学习实验，验证模型在不同任务间的适应性。通过调节α，观察模型在性能和计算成本上的折中表现，验证架构的灵活性。
Results
在T5-XXL中，LUMEN以三分之一的动态层实现了57.1%的EM，接近FiD，优于MemoryFiD约10%。不同模型规模中，LUMEN在相同FLOPs下性能提升超过20%。迁移学习显著改善在新任务上的表现，尤其在WebQuestions等小数据集上效果更佳。整体显示，少量动态层即可达到接近FiD的性能，模型越大，动态层比例越少，效率越高。
Applications
该架构适用于大规模知识问答、智能助手、企业知识库等场景。只需预存知识库表示，结合少量动态调整，即可实现高效推理，降低硬件成本，提升响应速度。未来可扩展到多模态信息融合、实时知识更新等，推动智能系统的广泛应用。
Limitations & Outlook
存储成本仍较高，尤其在超大知识库中，存储和检索开销大。动态层比例需调节，调参复杂。在知识快速变化的场景下，频繁重建存储表示可能抵消效率优势。未来需优化存储压缩和表示更新机制，提升实用性。
Plain Language Accessible to non-experts
想象你在一个图书馆里，有很多书（知识库），你可以提前把重要内容整理成索引卡（存储表示），这样查找时不用每次都翻阅整本书。但如果遇到特别的问题，你可以用一台快速扫描仪（动态编码器）现场调整索引，使它更符合你当前的问题。这样，你既不用每次都翻遍所有书，也能快速找到答案。LUMEN就像这样：提前准备好大部分信息，只在必要时做少量调整，既节省时间，又保证答案的准确性。
ELI14 Explained like you're 14
想象你在学校的图书馆里，有很多书（知识库），平时你提前把一些重要的内容做成索引卡（存储表示），这样平时查找就很快了。但是如果遇到特别难的问题，你可以用一台特别快的扫描仪（动态编码器）现场调整索引，让它更适合你的问题。这样，你既不用每次都翻遍所有书，也能很快找到答案。LUMEN就像这样聪明，它提前准备好大部分信息，然后只在必要时做少量调整，既省时间，又能得到准确答案。
Glossary
检索增强（Retrieval-Augmented）
利用外部知识库检索相关信息以增强模型能力。技术上结合检索机制与生成模型，提升知识密集任务表现。
论文中介绍的模型架构依赖检索增强策略。
预存表示（Pre-computed Representation）
提前对知识库内容编码并存储的向量表示，用于快速检索。减少实时编码成本，但缺乏动态条件化。
Memory方案的核心技术。
动态编码（On-the-fly Encoding）
在推理时根据输入实时生成表示，增强表示的条件化和适应性。成本较高，但性能更优。
LUMEN中少量层动态更新机制。
FLOPs（浮点运算次数）
衡量模型计算复杂度的指标，表示模型每次推理所需的浮点运算总数。
用于比较不同模型的效率。
Exact Match（精确匹配）
评估问答模型准确率的指标，统计模型输出与标准答案完全一致的比例。
模型性能的主要衡量标准。
Open Questions Unanswered questions from this research
- 1 如何进一步降低存储成本，尤其在超大语料库中实现高效索引和检索？
- 2 动态更新机制在极端场景（如实时知识变化）中的适应性和稳定性如何保障？
Applications
Immediate Applications
企业知识库问答系统
利用LUMEN快速检索企业内部知识，提升客服响应速度，降低硬件投入。
智能助手优化
在智能助手中实现高效知识检索，改善用户体验，减少响应延迟。
Long-term Vision
多模态知识融合
结合图像、视频等多源信息，构建全方位知识问答系统，推动智能交互发展。
Abstract
Retrieval-augmented language models such as Fusion-in-Decoder are powerful, setting the state of the art on a variety of knowledge-intensive tasks. However, they are also expensive, due to the need to encode a large number of retrieved passages. Some work avoids this cost by pre-encoding a text corpus into a memory and retrieving dense representations directly. However, pre-encoding memory incurs a severe quality penalty as the memory representations are not conditioned on the current input. We propose LUMEN, a hybrid between these two extremes, pre-computing the majority of the retrieval representation and completing the encoding on the fly using a live encoder that is conditioned on the question and fine-tuned for the task. We show that LUMEN significantly outperforms pure memory on multiple question-answering tasks while being much cheaper than FiD, and outperforms both for any given compute budget. Moreover, the advantage of LUMEN over FiD increases with model size.