WindVE: Collaborative CPU-NPU Vector Embedding
WindVE leverages CPU-NPU collaboration with linear regression-based queue management to boost concurrent vector embedding by 22.3%, reducing costs and improving throughput.
Key Findings
Methodology
This work analyzes deployment costs of vector embedding in inference services, establishing a mathematical model that highlights the importance of increasing maximum concurrency. A queue manager employing linear regression estimates optimal queue depths, enabling peak query offloading to CPU-NPU systems. WindVE system architecture exploits performance differences between CPU and NPU to efficiently handle traffic surges. Experiments demonstrate a 22.3% increase in concurrency, 1.22× throughput, and 18.6% cost savings, validating the approach’s effectiveness.
Key Results
- WindVE achieves up to 22.3% higher concurrency than baseline without offloading, maintaining SLO compliance and significantly improving throughput.
- The linear regression model accurately predicts maximum concurrency, guiding dynamic queue depth adjustments.
- Cost analysis shows 18.6% savings in deployment expenses, confirming the system’s economic benefits.
Significance
This research addresses the critical bottleneck in large-scale LLM inference—maximizing concurrency to reduce latency and costs. By intelligently offloading peak loads to CPU, it enhances hardware utilization, enabling scalable, cost-effective deployment of retrieval-augmented generation systems. The approach offers a practical solution for industry applications like search engines, chatbots, and content recommendation platforms, where response speed and cost efficiency are paramount.
Technical Contribution
The core innovation lies in integrating a linear regression-based estimator into a queue management framework for heterogeneous CPU-NPU systems. This allows real-time, data-driven adjustment of queue depths, optimizing resource utilization while respecting latency constraints. The system design effectively balances load between devices, surpassing traditional static scheduling methods, and provides a theoretical basis for maximum concurrency prediction under SLO constraints.
Novelty
This is the first work to systematically combine linear regression modeling with dynamic queue management for heterogeneous hardware in vector embedding tasks. Unlike prior approaches focusing solely on hardware acceleration, this method emphasizes intelligent scheduling to maximize concurrency and cost-efficiency, filling a research gap in scalable, adaptive inference systems.
Limitations
- The linear model assumes a linear relationship between latency and concurrency, which may not hold under extreme loads or diverse hardware configurations.
- Experimental validation is limited to specific hardware setups, requiring further testing across different platforms for generalization.
- The current system does not incorporate energy consumption or multi-task scheduling considerations, which are important for real-world deployment.
Future Work
Future research will explore non-linear models for more accurate concurrency prediction, incorporate multi-objective optimization including energy efficiency, and extend the framework to multi-task scenarios. Additionally, adaptive learning techniques could be integrated to dynamically refine scheduling policies based on real-time performance feedback.
AI Executive Summary
In the realm of large language model inference, vector embedding serves as a critical component for retrieval-augmented generation (RAG). However, the high latency associated with embedding and retrieval processes—up to 20% of total inference time—poses a significant challenge for scalable deployment. Traditional hardware configurations often underutilize CPU resources, which remain idle during intensive GPU/NPU tasks, leading to inefficiencies and higher costs.
This paper introduces WindVE, a novel system that exploits the heterogeneity of CPU and NPU architectures to improve concurrency and reduce deployment costs. The core idea is to offload peak query loads from NPU/GPU to CPU, which is underutilized, by employing a sophisticated queue management strategy. The system uses a linear regression model to estimate optimal queue depths dynamically, ensuring that latency constraints (SLOs) are maintained while maximizing throughput.
The architecture comprises a device detector, a queue manager, and a scheduler that prioritizes device utilization based on current load and device capabilities. During high traffic, excess queries are rerouted to CPU instances, effectively extending the system’s maximum concurrency. Experimental results on Huawei’s hardware platform show that WindVE achieves a 22.3% increase in concurrency, a 1.22× boost in throughput, and reduces deployment costs by 18.6%. These improvements demonstrate the system’s potential to significantly enhance the efficiency of large-scale inference services.
By intelligently balancing workload across heterogeneous devices, WindVE addresses the critical bottleneck of maximum concurrency, paving the way for more cost-effective, scalable AI inference solutions. Future work aims to incorporate adaptive, non-linear models and multi-objective optimization to further refine scheduling strategies, broadening its applicability across diverse hardware environments and application scenarios.
Deep Analysis
Background
随着大规模预训练模型的兴起，向量嵌入成为信息检索和问答系统的核心技术。早期工作如FlagEmbedding推动了嵌入效率，但在高维空间压缩和多硬件支持方面仍存在瓶颈。硬件方面，GPU和NPU的异构架构逐渐普及，提升推理性能，但调度策略多为静态配置，难以应对突发流量。传统方法多关注单一硬件优化，缺乏系统性调度策略，难以满足工业级高并发需求。
Core Problem
当前向量嵌入推理中，延迟高和资源利用率低限制了系统最大并发能力，导致成本上升和响应不稳定。特别在突发流量场景下，硬件资源不能充分发挥，影响用户体验和企业竞争力。现有调度多为静态配置，难以动态适应负载变化，亟需一种高效调度机制以提升最大并发和资源利用率。
Innovation
本文提出基于线性回归的队列深度估算模型，结合CPU-NPU异构架构，实现峰值查询卸载。创新点包括：1）设计智能队列管理器，动态调节队列深度；2）利用线性模型快速估算最大并发，避免繁琐调优；3）实现设备检测与调度策略优化，充分利用硬件性能差异。这些创新突破了传统调度瓶颈，显著提升系统吞吐和成本效率。
Methodology
- �� 设备检测：识别所有可用硬件（CPU、NPU）
- �� 队列建立：为每个设备配置独立队列，设置阈值（Cmax_NPU、Cmax_CPU）
- �� 请求调度：优先分配请求至NPU/GPU，超出阈值后卸载至CPU
- �� 压力测试：采集延迟数据，拟合线性关系（t = α·C + β）
- �� 最大并发估算：利用线性模型预测在SLO范围内的最大并发
- �� 峰值卸载：检测到超出阈值时，将请求卸载到CPU
- �� 动态调优：持续监控性能指标，调整队列阈值和调度策略
Experiments
在华为硬件平台上进行测试，比较未卸载方案，评估最大并发、吞吐和延迟。模拟突发流量，验证WindVE在满足SLO的同时最大化并发和吞吐。参数调优包括队列阈值和线性模型拟合，确保模型适应不同硬件性能差异。多轮压力测试验证模型鲁棒性。
Results
WindVE在测试中实现最高22.3%的并发提升，满足SLO，优于传统调度方案。吞吐提升至原来的1.22倍，成本节省18.6%。线性模型准确预测最大并发，调度策略有效缓解突发流量带来的延迟波动。系统表现稳定，适用性强。
Applications
适用于大规模企业级LLM推理服务，特别在检索增强生成（RAG）场景中，通过优化硬件调度提升响应速度和成本效益。广泛应用于搜索引擎、智能客服、内容推荐等行业，帮助企业降低成本，提升用户体验。未来可结合云端调度，实现全球资源优化。
Limitations & Outlook
模型假设线性关系在极端负载下可能失效，硬件性能差异大时调优复杂。实验环境受限于特定硬件配置，泛化需调参。未考虑多任务调度和能耗优化，未来需结合多目标策略提升系统整体性能和能效。
Plain Language Accessible to non-experts
想象你在一个厨房准备饭菜，平时用大锅煮饭，但在忙碌时大锅会变得太满，等待时间变长。于是，厨师用微波炉帮忙，把一些饭先放进去加热，这样可以同时做更多饭，大家都能快点吃到。WindVE就像这个厨房，利用不同的厨具（CPU和NPU）合作，把工作分配得更合理，让饭更快做好，花的钱也少一些。这样一来，厨房效率提高，饭菜更快端上桌，成本也降低了。
ELI14 Explained like you're 14
想象你在学校食堂吃饭，平时厨师用大锅煮饭，但如果很多人同时点饭，锅就会太满，等饭时间就会变长。于是，厨师用微波炉帮忙，把一部分饭先加热，这样就能同时做更多饭，大家都能快点吃到。WindVE就像这个厨房，用两种不同的厨具（CPU和NPU）合作，把工作分得更聪明，让饭更快做好，也更便宜。这样，大家都能更快吃到热腾腾的饭，而且花的钱也少一些。
Abstract
Retrieval-Augmented Generation is a technology that enhances large language models by integrating information retrieval. In the industry, inference services based on LLMs are highly sensitive to cost-performance ratio, prompting the need for improving hardware resource utilization in the inference service. Specifically, vector embedding and retrieval processes take up to 20% of the total latency. Therefore, optimizing the utilization of computational resources in vector embeddings is crucial for enhancing the cost-performance ratio of inference processes, which in turn boosts their product competitiveness.In this paper, we analyze the deployment costs of vector embedding technology in inference services, propose a theoretical formula, and determine through the mathematical expression that increasing the capacity to process concurrent queries is the key to reducing the deployment costs of vector embeddings. Therefore, in this paper, we focus on improving the product's capability to process concurrent queries. To optimize concurrency without sacrificing performance, we have designed a queue manager that adeptly offloads CPU peak queries. This manager utilizes a linear regression model to ascertain the optimal queue depths, a critical parameter that significantly influences the efficacy of the system. We further develop a system named WindVE that uses a CPU-NPU heterogeneous architecture to offload peak concurrent queries, which leverages the performance differences between the two processors to effectively manage traffic surges. Through experiments, we compare WindVE to the state-of-the-art vector embedding framework FlagEmbedding, and achieve a concurrency level up to 22.3% higher than the scheme without offloading.