Learning Retrieval Models with Sparse Autoencoders
SPLARE introduces a sparse latent retrieval model using pretrained SAEs, outperforming vocabulary-based methods in multilingual and out-of-domain tasks.
Key Findings
Methodology
This work integrates pretrained Sparse Autoencoders (SAEs) into large language models (LLMs), extracting sparse latent features from intermediate transformer layers to replace traditional vocabulary projections. The training employs contrastive distillation with a KL divergence loss, combined with FLOPS-based sparsity regularization. Only the linear projection head is fine-tuned, keeping SAE parameters frozen, which simplifies training and enhances generalization. Top-K pooling controls sparsity levels, balancing efficiency and effectiveness. The models are evaluated on multilingual datasets like MMTEB, demonstrating superior performance over vocabulary-based SPLADE models, especially in cross-lingual and domain-specific scenarios.
Key Results
- The SPLARE-7B model achieves top results on MMTEB multilingual and English retrieval tasks, with an average nDCG@10 improvement of 2.4 points over SPLADE-Llama, using only 2B parameters, with faster inference and reduced storage.
- In multilingual and out-of-domain evaluations, SPLARE models show strong generalization, outperforming vocabulary-based models by approximately 1.8 points, especially in low-resource languages and specialized domains like law and medicine.
- Ablation studies reveal that features extracted from intermediate layers (~2/3 of model depth) yield optimal retrieval performance. Increasing the SAE width improves effectiveness logarithmically, supporting scalable high-dimensional sparse representations up to millions of features.
Significance
This research addresses the limitations of fixed vocabulary-based retrieval models by leveraging sparse latent features, enabling robust, scalable, and multilingual information retrieval. It enhances semantic structure and interpretability, reduces dependency on vocabulary size, and supports efficient indexing and search in large-scale, multilingual, and multimodal environments. The approach paves the way for more flexible and generalizable retrieval systems, crucial for real-world applications like search engines, digital assistants, and knowledge bases.
Technical Contribution
The paper pioneers the integration of large-scale pretrained SAEs into the latent space of LLMs for retrieval, proposing a novel framework that combines intermediate layer feature extraction, sparsity regularization, and minimal fine-tuning. This approach supports extremely high-dimensional sparse vectors, surpassing traditional vocabulary limitations, and provides theoretical and practical advantages in efficiency, scalability, and multilinguality. The methodology demonstrates that high-quality sparse representations can be learned and effectively utilized for retrieval tasks without extensive retraining or data augmentation.
Novelty
This is the first work to embed pretrained SAEs into the transformer’s intermediate layers for multilingual sparse retrieval, replacing fixed vocabulary projections with a scalable, semantically rich latent space. Unlike prior models relying solely on vocabulary embeddings, SPLARE exploits the potential of learned latent features, enabling scalable, language-agnostic, and multimodal retrieval with minimal task-specific training.
Limitations
- The model underperforms in highly specialized domains like code retrieval, indicating a need for domain-specific SAE training. Sparse regularization parameters require careful tuning, which can be complex and dataset-dependent. Large-scale SAE training incurs high computational costs, limiting widespread adoption. Future work should focus on automating sparsity control, domain adaptation, and reducing training costs.
Future Work
Future directions include integrating multimodal data (images, audio) into the latent space, developing adaptive sparsity regularization methods, and exploring external knowledge integration to enrich semantic features. Additionally, scaling the approach to even larger models and more diverse languages, as well as optimizing inference speed and index structures, will further enhance practical deployment.
AI Executive Summary
The rapid evolution of large language models (LLMs) has revolutionized information retrieval, yet traditional vocabulary-based approaches face significant limitations in scalability, multilinguality, and interpretability. Fixed vocabularies restrict the models’ ability to generalize across languages and domains, especially in low-resource settings. To address these challenges, this paper introduces SPLARE, a novel retrieval framework leveraging pretrained Sparse Autoencoders (SAEs) to generate high-dimensional, sparse latent features. Unlike conventional methods that project input sequences onto a fixed vocabulary, SPLARE extracts sparse representations from intermediate transformer layers, forming a flexible, language-agnostic feature space.
The core idea is to replace the vocabulary projection head with a pretrained SAE, which decomposes dense token representations into a set of interpretable, sparse latent features. During training, the authors employ a contrastive distillation approach, where a teacher model guides the sparse embeddings, and sparsity is enforced via FLOPS regularization. Only the linear projection layer is fine-tuned, while the SAE remains frozen, simplifying training and improving generalization. Top-K pooling further controls sparsity, balancing retrieval effectiveness and efficiency.
Experimental results demonstrate that SPLARE models, especially the 7B-parameter version, outperform existing vocabulary-based models like SPLADE across multiple benchmarks, including MMTEB and multilingual datasets. The models exhibit remarkable robustness in cross-lingual and domain-specific retrieval, with significant improvements in low-resource languages and specialized fields such as law and medicine. Ablation studies reveal that features from intermediate transformer layers yield optimal performance, and increasing SAE width enhances effectiveness logarithmically.
This work represents a significant advancement in scalable, multilingual, and interpretable retrieval systems. By moving beyond fixed vocabularies and harnessing the potential of learned latent features, SPLARE offers a flexible, efficient, and powerful approach suitable for large-scale real-world applications. Future research will explore multimodal extensions, adaptive sparsity control, and external knowledge integration, further broadening its impact.
Deep Analysis
Background
Recent progress in deep learning has transformed information retrieval,从BERT到CLIP等模型推动了密集表示的发展。然而，词汇表限制了多语种和多模态场景的扩展，导致模型在跨语种和专业领域表现不足。稀疏检索模型如SPLADE通过词汇空间实现高效索引，但受限于固定词表，难以应对多样化需求。预训练SAEs在多语种、多模态任务中表现出强大潜力，但尚未广泛应用于检索。本文结合两者优势，提出潜在空间稀疏检索，旨在突破词汇空间的局限。
Core Problem
传统词汇空间检索模型依赖固定词表，难以扩展到多语种和多模态场景，且在低资源语言和专业领域表现不足。密集表示虽性能优越，但存储和计算成本高，难以实现大规模高效索引。如何利用潜在空间的稀疏特征，兼顾效率和泛化能力，成为亟待解决的问题。核心在于设计高效、可扩展的潜在特征表示，确保其在多语种、多领域中的表现一致。
Innovation
本研究的创新点包括：1）引入预训练SAEs到Transformer中间层，提取稀疏潜在特征，替代词汇空间；2）设计稀疏正则化（如FLOPS）与Top-K池化，控制稀疏度，提升效率；3）只微调线性投影头，简化训练流程，增强泛化能力；4）支持百万级潜在特征维度，突破词表限制，提升多语种、多模态检索能力。这些创新解决了词汇空间的局限，提供了更丰富的语义表达。
Methodology
- �� 选择预训练的SAEs（如Llama Scope）在Transformer中间层提取稀疏特征。• 利用稀疏自编码器（Encoder+Decoder）将密集表示转化为高维稀疏向量。• 采用对比蒸馏训练，保持模型性能，同时引入FLOPS正则化确保稀疏性。• 只微调线性投影头，冻结SAE参数，简化训练流程。• 在训练中引入Top-K池化调节稀疏度，确保索引效率。• 利用倒排索引结构支持稀疏向量的高效检索。• 通过多语种、多领域数据进行微调，增强模型泛化能力。
Experiments
在MMTEB、多语种和专业领域数据集上进行评估，比较SPLARE与词汇空间模型（如SPLADE）的性能差异。采用nDCG@10、Recall等指标，调节潜在空间宽度和稀疏正则化参数。进行多层次消融，分析不同层和宽度对性能的影响。模型训练在大规模多语种数据上进行，采用LoRA微调，确保泛化能力。实验还验证了稀疏正则化和Top-K池化对效率和效果的影响。
Results
SPLARE-7B在MMTEB多语和英语任务中表现优异，平均nDCG@10提升2.4点，参数量仅为2B，推理速度快，存储空间减少70%。在低资源语言和专业领域表现优于词汇模型，提升幅度达1.8点。中间层（约深度的2/3）提取的特征最优，宽度越大，性能越好，支持百万维潜在空间。模型的稀疏性调节灵活，能在保持效果的同时大幅提升检索效率。
Applications
该模型适用于多语种搜索引擎、跨模态信息检索、专业领域知识库检索等场景。只需少量参数调节，便能实现高效索引和快速检索，尤其适合大规模多语种、多模态环境。未来可结合知识图谱，丰富潜在特征的语义信息，提升系统的智能化水平。
Limitations & Outlook
模型在极端专业场景（如特定代码或高度领域化文本）表现有限，需训练专用SAE。稀疏正则化参数调节复杂，可能影响模型稳定性。大规模SAE训练成本高，限制其普及。未来需优化稀疏控制策略，提升模型的适应性和效率。
Abstract
Sparse autoencoders (SAEs) provide a powerful mechanism for decomposing the dense representations produced by Large Language Models (LLMs) into interpretable latent features. We posit that SAEs constitute a natural foundation for Learned Sparse Retrieval (LSR), whose objective is to encode queries and documents into high-dimensional sparse representations optimized for efficient retrieval. In contrast to existing LSR approaches that project input sequences into the vocabulary space, SAE-based representations offer the potential to produce more semantically structured, expressive, and language-agnostic features. Building on this insight, we introduce SPLARE, a method to train SAE-based LSR models. Our experiments, relying on recently released open-source SAEs, demonstrate that this technique consistently outperforms vocabulary-based LSR in multilingual and out-of-domain settings. SPLARE-7B, a multilingual retrieval model capable of producing generalizable sparse latent embeddings for a wide range of languages and domains, achieves top results on MMTEB's multilingual and English retrieval tasks. We also developed a 2B-parameter variant with a significantly lighter footprint.