InSituANN: Revisiting IVF for PCIe-Efficient Billion-Scale Vector Search
InSituANN leverages IVF with host memory to enable billion-scale vector search on a single GPU, avoiding PCIe bottlenecks and achieving high throughput.
Key Findings
Methodology
This paper introduces InSituANN, a heterogeneous IVF-based system that performs coarse search on GPU and fine search in host memory, eliminating large vector transfers over PCIe. It employs GPU-based compact routing and optional pruning, maintaining IVF simplicity. The system features a novel ultra-fast IVF construction pipeline, leveraging GPU acceleration for training and assignment, reducing build time from hours to minutes. Stage-wise optimizations, PQ candidate reduction, and SIMD-accelerated exact distance computations enable high throughput and recall at billion scale, with minimal storage overhead.
Key Results
- On SIFT-1B, InSituANN constructs the index in 5.2 minutes, ~350× faster than HNSW’s 30.4 hours. Query throughput at matched recall exceeds baseline by 104.9× to 4298.2×, outperforming DiskANN and Rummy significantly.
- It maintains high recall (>90%) while drastically reducing index construction time and storage costs, demonstrating practical scalability.
- The combination of stage-aware placement, PQ pruning, and GPU-CPU pipeline yields a system capable of billion-scale retrieval with unprecedented efficiency.
Significance
This work addresses the critical bottleneck of PCIe bandwidth in GPU-based billion-scale vector search. By intelligently partitioning tasks between GPU and host memory, it enables high-speed, cost-efficient retrieval on commodity hardware. The approach significantly advances the scalability of vector databases, facilitating real-time large-scale recommendation, semantic search, and multimodal retrieval. Its practical deployment potential makes it a milestone in AI infrastructure, reducing hardware costs while boosting performance.
Technical Contribution
The paper introduces a novel heterogeneous IVF architecture that separates coarse routing and fine search across GPU and host, respectively. It innovates with ultra-fast IVF construction leveraging GPU training and assignment, and employs PQ-based candidate pruning combined with SIMD-optimized distance evaluation. These innovations collectively enable billion-scale search on a single GPU, with minimal PCIe data transfer, high accuracy, and low latency, setting new standards in large-scale vector retrieval.
Novelty
This is the first comprehensive system to realize billion-scale vector search on a single GPU by combining stage-aware task placement, ultra-fast IVF construction, and PQ candidate reduction. Unlike prior graph-based methods, it maintains IVF's simplicity and update-friendliness while achieving orders-of-magnitude speedup. Its architecture fundamentally redefines the hardware-software boundary in large-scale vector search, making high-performance retrieval practical on commodity hardware.
Limitations
- The system assumes static datasets; dynamic updates require frequent index rebuilds, impacting real-time applicability.
- High-dimensional or extremely large datasets still pose challenges in storage and retrieval efficiency, especially under limited hardware resources.
- Current focus is on static, single-GPU setups; multi-GPU scaling and real-time updates are future directions.
Future Work
Future efforts will explore dynamic index updates, multi-GPU scalability, and multi-modal data integration. Developing adaptive indexing strategies and real-time update mechanisms will further enhance system flexibility and applicability in evolving data environments.
AI Executive Summary
As data volumes grow exponentially, the demand for efficient large-scale vector search intensifies, especially in recommendation systems, semantic retrieval, and large language models. Traditional GPU-based methods excel in parallelism but are hampered by limited VRAM, making it difficult to handle billion-scale datasets entirely in GPU memory. Existing solutions like graph-based indexes (HNSW, DiskANN) face challenges with irregular memory access patterns, high storage overhead, and expensive updates. To address these limitations, this work proposes InSituANN, a novel heterogeneous IVF system that divides the search process across GPU and host memory.
InSituANN performs the coarse search—routing queries to relevant clusters—entirely on GPU using dense matrix operations, leveraging GPU Tensor Cores for speed. The fine search, which involves scanning large host-resident inverted lists, is executed in situ on the CPU, avoiding costly PCIe transfers of high-dimensional vectors. To further reduce candidate evaluation costs, residual PQ codes are used for candidate pruning, with SIMD-accelerated distance computations ensuring high efficiency.
The system's ultra-fast IVF construction pipeline employs GPU training and assignment, reducing index building time from hours to minutes. Experimental results on datasets like SIFT-1B show that InSituANN constructs the index in just 5.2 minutes, about 350× faster than HNSW. Query throughput at high recall surpasses baseline methods by over 100×, reaching thousands of queries per second while maintaining high accuracy.
This architecture significantly lowers hardware costs and enhances scalability, making billion-scale vector retrieval practical on commodity hardware. It opens new avenues for real-time large-scale retrieval applications and sets a new benchmark for efficiency and simplicity in vector search systems. Future work will focus on dynamic updates, multi-GPU scaling, and multi-modal data integration, further broadening its impact.
Deep Analysis
Background
近年来，随着深度学习和大数据技术的发展，向量检索成为信息检索、推荐系统和大模型的重要支撑。早期代表性方法如HNSW、IVF和图结构索引在高召回率方面表现优异，但在亿级数据规模下，存储和计算成本急剧上升。GPU的高并行能力被广泛应用于向量检索，但受限于显存容量，难以支持全驻留。传统方案多依赖图结构索引，存在存储冗余、更新复杂和扩展性差的问题。近年来，IVF索引因其结构简单、易于部署，成为研究热点，但在大规模场景中，PCIe带宽瓶颈限制了GPU的应用。本文在此背景下，提出一种结合GPU高速路由和主机存储的异构架构，突破了现有瓶颈，推动亿级向量检索的实用化。
Core Problem
核心问题在于如何在单GPU环境下实现亿级向量的高效检索。传统GPU索引受显存限制，难以存储全部基向量，导致频繁的PCIe数据传输，严重制约吞吐量。即使采用主存存储，GPU在细搜索阶段仍需大量数据传输，造成延迟和带宽瓶颈。此外，索引构建时间长、存储空间大、更新困难，限制了工业应用的推广。解决这一问题需要设计一种在硬件资源有限条件下，兼顾速度、存储和更新的索引架构。
Innovation
本研究的创新点包括：1）提出基于IVF的异构架构，将粗搜索在GPU上完成，细搜索在主机中执行，避免大规模基向量的PCIe传输瓶颈；2）设计超快的索引构建流程，利用GPU加速训练和分配，将构建时间缩短至5.2分钟；3）引入PQ候选缩减机制，通过GPU扫描紧凑的PQ码，减少候选数，提高筛选效率；4）采用SIMD优化的距离核，提升精确距离验证速度。这些创新使系统在保持高召回的同时，实现了极高的吞吐率和存储效率。
Methodology
- �� 预训练IVF索引：利用K-means对向量进行聚类，生成簇中心。• 粗搜索：GPU端进行批量距离计算，快速筛选出最接近的簇。• 索引训练：GPU加速训练簇中心，快速完成索引训练和分配。• 细搜索：CPU端在主机内存中扫描选中的倒排列表，计算精确距离。• PQ候选缩减：在GPU上扫描紧凑的PQ码，筛选候选集。• 最终排序：在CPU端利用SIMD加速的距离核，完成最终排序。• pipeline优化：GPU和CPU并行处理不同阶段，提升整体吞吐。
Experiments
采用SIFT-1B、DEEP-1B等公开大规模数据集，比较基线包括HNSW、DiskANN和Rummy。指标涵盖召回率、查询吞吐量和索引构建时间。调节probe数和候选数，验证不同配置的性能表现。多次实验显示，InSituANN在保持90%以上召回率时，索引构建时间缩短至5.2分钟，查询吞吐量提升数千倍，优于现有方法。还进行了消融实验，验证PQ候选缩减和GPU优化的贡献。
Results
在SIFT-1B数据集上，InSituANN索引构建时间仅为5.2分钟，比HNSW的30.4小时快约350倍。在查询性能方面，匹配召回率下，端到端吞吐量比PCIe瓶颈的Rummy提升104.9倍至4298.2倍，比DiskANN提升2.4倍至4.6倍。系统在保持高召回的同时，显著降低了存储成本，验证了其工业应用潜力。这些结果充分体现了异构架构在大规模向量检索中的优势。
Applications
该系统适用于大规模推荐、语义搜索、知识增强的检索任务，尤其适合硬件资源有限的场景。企业可利用其低成本、高效率的特性，部署在云端或边缘设备，实现实时大规模向量匹配。未来可结合多模态数据和动态索引更新，拓展应用范围。
Limitations & Outlook
系统在动态数据更新和多模态融合方面仍需优化，当前主要面向静态数据集。硬件依赖性较强，GPU资源有限时性能下降。此外，索引构建和维护仍存在一定成本，未来需进一步提升自适应能力和扩展性。
Plain Language Accessible to non-experts
想象你在一个巨大的仓库里整理各种商品，每个商品都有一个标签（向量）。如果你想找到和某个标签最相似的商品，传统方法就像逐个翻找，非常慢。现在，InSituANN就像用一个快速扫描仪先把仓库分成几个区域（簇），只在这些区域里找，避免了每次都要扫描整个仓库。然后，把每个区域的商品存放在仓库的不同角落（主机内存），在需要找东西时，只在这些角落里快速翻找，大大节省时间和交通。这样一来，不管仓库多大，也能快速找到想要的商品，而且花费很少。这就像用智能导航在超市里找到心仪商品一样简单。
ELI14 Explained like you're 14
想象你在学校图书馆找一本书，书很多，光用眼睛看一遍很慢。以前的方法就像你一个一个翻书页，花费很长时间。现在，有个聪明的机器人帮你先用扫描仪快速划出几个可能的区域（簇），只在这些区域里找书。这个机器人还把每个区域的书都存放在不同的架子上（主机内存），你只需要在这些架子上快速翻找，就不用每次都跑到仓库的每个角落去找。这样一来，不管图书馆有多大，你都能很快找到想要的书，而且不需要花很多时间和力气。这就像用智能导航帮你在大图书馆里快速找到目标书一样，既省时间又省力。
Abstract
Approximate nearest neighbor search (ANNS) over billion-scale vector datasets has become a foundational operator for modern retrieval systems, powering large-scale recommendation, semantic search, and LLM/RAG workloads. Although GPUs offer massive parallelism and high-bandwidth memory for batched vector search, their limited VRAM capacity makes fully GPU-resident billion-scale indexes difficult to deploy. In CPU-GPU heterogeneous designs, keeping the base vectors in host memory avoids this capacity limit, but naively offloading fine search to the GPU introduces a new bottleneck: large volumes of base-vector data must be streamed over PCIe. We present InSituANN, an IVF-based ANNS engine that enables billion-scale vector search on a single commodity GPU. InSituANN keeps original base vectors in host memory, performs fine search in situ, and uses the GPU for compact routing and optional pruning. As a result, query processing avoids PCIe transfers of high-dimensional base vectors while retaining the simplicity of IVF. Beyond query performance, we further design an ultra-fast IVF construction path for InSituANN. On SIFT-1B, InSituANN builds the IVF index in 5.2 minutes, about 350x faster than the measured 30.4-hour HNSW build. At matched recall on billion-scale datasets, InSituANN improves end-to-end throughput by 104.9x-4298.2x over the PCIe-bound Rummy baseline and by 2.4x-4.6x over DiskANN on SIFT-1B and DEEP-1B. Together with strong recall-throughput trade-offs and lower index space than graph-based alternatives, these gains make billion-scale retrieval practical on cost-efficient hardware. We open-source InSituANN at https://github.com/mindtravel/InSituANN-OpenSource.