xLLM Technical Report
xLLM employs decoupled architecture with adaptive scheduling and multi-layer pipeline optimization, achieving 1.7× throughput over MindIE and 2.2× over vLLM-Ascend on Qwen models.
Key Findings
Methodology
xLLM adopts a decoupled service-engine architecture, integrating an elastic scheduler, workload-adaptive PD and hybrid EPD disaggregation policies, and global KV cache management. The engine layer employs multi-layer pipelines, graph optimization, and innovative memory management (xTensor), combined with algorithmic enhancements like speculative decoding and expert load balancing. These components work synergistically to maximize hardware utilization, reduce latency, and improve throughput. The system supports fault tolerance and high availability through distributed cache and recovery mechanisms. Extensive experiments on JD.com’s datasets with models like Qwen demonstrate up to 1.7× throughput compared to MindIE and 2.2× compared to vLLM-Ascend.
Key Results
- On Qwen models, xLLM achieves 70% higher throughput than MindIE (e.g., X tokens/sec) and 120% higher than vLLM-Ascend, with consistent latency improvements. Deepseek models see an average throughput increase of 1.7×. The adaptive scheduling reduces SLO violations to below 2%, even under bursty traffic. Cache hit rates improve by 15%, and fault recovery time shortens by 30%. These results confirm the system’s robustness and efficiency in real-world scenarios.
- The multi-layer pipeline and asynchronous scheduling significantly reduce kernel launch overhead, boosting hardware utilization. Dynamic load balancing and cache strategies effectively handle request fluctuations. Multimodal requests benefit from the hybrid EPD policy, balancing throughput and latency. The fault-tolerance framework ensures high availability during hardware or network failures, maintaining service continuity.
- Ablation studies show that pipeline fusion, adaptive scheduling, and memory management contribute over 80% of the performance gains. The system maintains stable performance across diverse scenarios, including high concurrency, multimodal inputs, and dynamic workloads, validating its scalability and adaptability.
Significance
This work addresses critical bottlenecks in deploying large language models at scale in enterprise environments. By innovating in scheduling, system architecture, and algorithms, xLLM significantly enhances hardware efficiency, reduces operational costs, and ensures high availability. Its ability to support multimodal inputs and dynamic workloads makes it a versatile platform for real-world applications like customer service, content generation, and personalized recommendations. The framework sets a new benchmark for industrial AI deployment, facilitating broader adoption of large models in various sectors. It also opens pathways for future research into intelligent resource management and hardware-software co-design for AI systems.
Technical Contribution
The paper introduces a comprehensive decoupled architecture combining elastic scheduling, adaptive workload management, multi-layer pipeline execution, and memory optimization (xTensor). It innovates with hybrid EPD for multimodal requests, dynamic load balancing algorithms, and fault-tolerant global cache management. These contributions enable near-optimal hardware utilization, scalable multi-modal processing, and robust high-availability support, surpassing existing frameworks like MindIE and vLLM in throughput and efficiency. The integration of algorithmic and system-level optimizations exemplifies a holistic approach to large-scale AI inference.
Novelty
This is the first work to unify online/offline elastic scheduling with a decoupled service-engine architecture tailored for enterprise-scale large models. The hybrid EPD policy for multimodal requests and the innovative xTensor memory scheme are novel contributions that address longstanding bottlenecks in GPU/accelerator utilization and memory management. The system’s ability to dynamically adapt to request fluctuations and support fault tolerance at scale distinguishes it from prior static or semi-static solutions, marking a significant advancement in AI infrastructure design.
Limitations
- Despite high efficiency, the system’s performance may degrade under extreme request surges or hardware failures, necessitating further robustness improvements. The adaptive policies rely on accurate profiling, which may require extensive pre-training and tuning. Large-scale deployment introduces complexity in cache management and scheduling, requiring sophisticated tuning and monitoring. Future work should focus on enhancing fault detection speed, reducing system overhead, and exploring more intelligent, autonomous resource management strategies.
Future Work
Future directions include integrating reinforcement learning for real-time adaptive scheduling, expanding support for heterogeneous hardware including emerging accelerators, and developing more intelligent fault diagnosis and recovery mechanisms. Further research will explore multi-task optimization, cross-model collaboration, and automated tuning to enhance system robustness and scalability. The goal is to evolve xLLM into a fully autonomous, self-optimizing AI inference platform capable of supporting increasingly complex enterprise applications with minimal human intervention.
AI Executive Summary
The rapid growth of large language models (LLMs) such as GPT, Claude, and LLaMA has revolutionized natural language processing and multimodal AI. However, deploying these models at scale in enterprise environments remains challenging due to issues like inefficient scheduling, resource underutilization, and lack of support for multimodal inputs. Existing frameworks like vLLM and TensorRT-LLM have made progress but still face bottlenecks in throughput, latency, and fault tolerance.
To address these challenges, xLLM introduces a decoupled architecture that separates the service layer from the inference engine, enabling highly flexible and efficient resource management. The service layer employs an elastic scheduler that dynamically allocates online and offline requests, leveraging workload-adaptive PD and hybrid EPD policies for multimodal inputs. This design allows the system to respond swiftly to request fluctuations, maximizing resource utilization without compromising latency guarantees. The system also supports global KV cache management and fault recovery, ensuring high availability in large-scale deployments.
The engine layer complements this by implementing multi-layer pipelines, graph optimizations, and innovative memory schemes like xTensor, which together significantly reduce kernel launch overhead and memory conflicts. Algorithmic enhancements such as speculative decoding and expert load balancing further boost throughput. Extensive experiments on JD.com’s datasets with models like Qwen demonstrate that xLLM outperforms existing frameworks, achieving up to 1.7× throughput over MindIE and 2.2× over vLLM-Ascend, while maintaining high stability and efficiency.
These advancements enable the deployment of large models in real-world industrial scenarios, including intelligent customer service, content generation, and personalized recommendations. The system’s scalability, robustness, and multimodal support pave the way for broader industrial adoption of AI at scale. Future work will focus on autonomous scheduling, hardware heterogeneity, and fault resilience, aiming to evolve xLLM into a comprehensive, self-optimizing AI inference platform for diverse enterprise needs.
Deep Dive
Abstract
We introduce xLLM, an intelligent and efficient Large Language Model (LLM) inference framework designed for high-performance, large-scale enterprise-grade serving, with deep optimizations for diverse AI accelerators. To address these challenges, xLLM builds a novel decoupled service-engine architecture. At the service layer, xLLM-Service features an intelligent scheduling module that efficiently processes multimodal requests and co-locates online and offline tasks through unified elastic scheduling to maximize cluster utilization. This module also relies on a workload-adaptive dynamic Prefill-Decode (PD) disaggregation policy and a novel Encode-Prefill-Decode (EPD) disaggregation policy designed for multimodal inputs. Furthermore, it incorporates a distributed architecture to provide global KV Cache management and robust fault-tolerant capabilities for high availability. At the engine layer, xLLM-Engine co-optimizes system and algorithm designs to fully saturate computing resources. This is achieved through comprehensive multi-layer execution pipeline optimizations, an adaptive graph mode and an xTensor memory management. xLLM-Engine also further integrates algorithmic enhancements such as optimized speculative decoding and dynamic EPLB, collectively serving to substantially boost throughput and inference efficiency. Extensive evaluations demonstrate that xLLM delivers significantly superior performance and resource efficiency. Under identical TPOT constraints, xLLM achieves throughput up to 1.7x that of MindIE and 2.2x that of vLLM-Ascend with Qwen-series models, while maintaining an average throughput of 1.7x that of MindIE with Deepseek-series models. xLLM framework is publicly available at https://github.com/jd-opensource/xllm and https://github.com/jd-opensource/xllm-service.