CoRun: Padding is Simple and Efficient for Deterministic LLM Inference
CoRun leverages position invariance to achieve deterministic, high-throughput LLM inference, boosting throughput by 15-324%.
Key Findings
Methodology
This work analyzes GPU kernel properties, distinguishing batch invariance from position invariance. CoRun employs scheduling strategies—isolated prefill and fixed-shape decoding—using CUDA graphs to control execution shapes. By binding random number generator states to requests, it ensures identical outputs across concurrent requests, positions, and arrival orders. The system’s design leverages the fact that most kernels are position-invariant, enabling high efficiency without full batch invariance. Extensive experiments on models like Qwen-3-235B, DeepSeek V3, and Hy3 validate the approach, showing significant throughput gains and output consistency.
Key Results
- On Qwen-3-235B-A22B, CoRun achieves deterministic outputs with throughput increases of 15-324%, reducing time-to-first-token by 51.8% and time-per-output-token by 48.6%. Similar improvements are observed on DeepSeek V3 and Hy3, with latency dropping from 88.4ms to 32.6ms for large models. Ablation studies confirm that request isolation, fixed-shape decoding, and request-bound RNG are critical for guarantees. The system maintains high GPU utilization and stability across scenarios, outperforming traditional batch-invariant kernels.
- Comparison between standard and batch-invariant kernels shows that the latter incurs up to 36x latency overhead in operators like MatMul and Attention, severely limiting throughput. CoRun’s approach avoids this overhead by exploiting position invariance, enabling faster execution while preserving output consistency. The experimental results demonstrate that the proposed scheduling and shape control strategies effectively balance performance and determinism.
- The core technical innovation involves controlling execution shape at each inference stage, using isolated prefill to maintain original prompt shapes, and fixed-shape CUDA graphs for decoding. Request-bound random seed states guarantee sampling reproducibility. These mechanisms collectively ensure identical token sequences regardless of request concurrency, position, or arrival order, validated through formal proofs and extensive empirical testing.
Significance
This research addresses a fundamental challenge in deploying large language models—ensuring deterministic outputs without sacrificing performance. By exploiting the inherent position invariance of many GPU kernels, CoRun provides a scalable, efficient solution that maintains output reproducibility crucial for model evaluation, debugging, and reinforcement learning. It bridges the gap between high throughput and deterministic inference, a long-standing trade-off in AI systems. The approach paves the way for more reliable, predictable AI services, fostering trust and facilitating rigorous scientific analysis. Its generality across architectures and models highlights its broad applicability, promising to influence future hardware-aware AI system designs.
Technical Contribution
CoRun introduces a novel scheduling framework that leverages position invariance to achieve deterministic inference without requiring full batch invariance. Key innovations include isolated prefill to preserve individual request shapes, fixed-shape CUDA graph decoding for efficiency, and request-bound RNG states for sampling consistency. The system formalizes a determinism contract, with rigorous proofs ensuring identical outputs across requests, positions, and arrival orders. This work extends the capabilities of GPU kernel optimization, enabling high-performance, deterministic large-scale inference, and opens new avenues for hardware-aware scheduling and kernel design in AI systems.
Novelty
This is the first work to exploit position invariance in GPU kernels for deterministic inference, bypassing the performance bottlenecks of full batch invariance. By controlling execution shapes through scheduling rather than kernel modifications, CoRun achieves both high throughput and output consistency. The integration of CUDA graphs with request-specific shape control and request-bound random seeds represents a significant innovation, setting a new standard for scalable, deterministic AI inference systems. This approach fundamentally differs from prior methods that rely solely on kernel invariance or static batching.
Limitations
- The approach assumes that most kernels are position-invariant; certain operators like tensor parallel reductions or non-standard kernels may still break guarantees. Handling these exceptions requires additional mechanisms.
- In multi-GPU setups, synchronization overhead and communication costs could impact scalability. The current design primarily targets single-GPU or tightly coupled environments.
- Request shape control relies on accurate scheduling and may face challenges with highly variable prompt lengths or dynamic workloads, potentially limiting flexibility in some scenarios.
Future Work
Future directions include extending the framework to multi-GPU environments with optimized synchronization, supporting more complex operators beyond position invariance, and integrating adaptive shape control for highly variable prompts. Further research could explore dynamic kernel adaptations and hardware modifications to broaden the applicability of deterministic inference. Additionally, automating the scheduling and shape management process could make the system more robust and easier to deploy across diverse hardware platforms.
AI Executive Summary
Large language models (LLMs) have revolutionized natural language processing, yet their deployment faces a persistent challenge: output nondeterminism. Even with fixed sampling parameters and seeds, GPU scheduling intricacies—such as dynamic batching—introduce variability in token outputs. This variability hampers model evaluation, debugging, and reinforcement learning, creating a barrier to reliable deployment. Traditional solutions rely on batch-invariant kernels, which enforce fixed computation patterns across shapes, but at a severe performance cost—doubling latency and reducing throughput by up to 74%. Such trade-offs are unacceptable for real-world applications demanding both speed and reproducibility.
In response, the authors propose CoRun, a novel scheduling-based system that exploits the inherent position invariance of most GPU kernels. Instead of making every kernel batch-invariant, CoRun controls the execution shape through request isolation during prefill and fixed-shape decoding during inference. This approach maintains the efficiency of optimized kernels while ensuring deterministic outputs. The system employs CUDA graphs to orchestrate execution, binds random seeds to requests, and manages request queues to prevent shape-dependent variability. Extensive experiments on models like Qwen-3-235B, DeepSeek V3, and Hy3 demonstrate that CoRun achieves output consistency with throughput improvements ranging from 15% to over 300%, significantly reducing latency and increasing GPU utilization.
This work marks a significant advance in scalable, reliable AI inference. By moving beyond the constraints of full batch invariance, it opens new avenues for deploying large models in production environments where speed and reproducibility are both critical. The methodology’s generality suggests broad applicability across diverse models and hardware platforms. Future work aims to extend multi-GPU support, incorporate more operators, and further optimize scheduling strategies, promising a new era of deterministic, high-performance AI systems.
Deep Analysis
Background
近年来，深度学习模型，尤其是大规模语言模型（LLM），在自然语言处理领域取得了巨大突破。代表性工作如GPT系列、BERT、LLaMA等，依靠大规模参数和数据训练，显著提升了模型性能。传统推理系统多采用批次调度以提高GPU利用率，但在多请求环境中，GPU调度的动态变化引入了随机性，影响输出的可重复性。现有研究多关注核函数的优化（如Transformer核、Attention核），但未充分解决GPU调度引入的非确定性问题。全批次不变核虽能保证输出一致，但严重牺牲性能，难以满足工业应用的需求。
Core Problem
核心问题在于GPU调度中的非确定性，特别是在动态批次环境下。请求到达时间、长度变化导致批次形状不断变化，影响核函数的执行顺序和浮点运算顺序，从而引入微小数值差异。这些差异在自回归生成中会被放大，导致生成结果不一致。这不仅影响模型评估的可靠性，也阻碍调试和RL训练的稳定性。现有方案要么牺牲性能实现全批次不变，要么接受随机性，亟需一种兼顾效率与确定性的方法。
Innovation
本文的创新点在于：1）利用核函数的“位置不变性”特性，控制请求的输入形状，避免全批次不变核的性能瓶颈；2）设计隔离预填充（prefill），保持每个请求的原始prompt形状；3）采用固定形状的CUDA图调度解码（decode），提升效率；4）请求绑定随机数状态，确保采样的一致性。这些创新突破了传统核优化的限制，实现了高效且确定的推理，兼顾性能和输出一致性，为大模型推理提供新思路。
Methodology
- �� 分析GPU核函数中的位置不变性与批次不变性，定义核函数的两类不变性属性。• 设计调度策略，将请求在预填充阶段隔离，保持请求原始形状。• 在解码阶段，采用固定形状的CUDA图调度，确保每个请求的输入形状一致。• 利用请求绑定的随机数状态，保证采样过程的确定性。• 结合请求队列管理，动态调度请求，避免批次形状变化带来的核函数调度差异。• 通过数学证明，确保在所有请求、位置、到达顺序下输出一致。• 实现系统在Qwen、DeepSeek等模型上，验证其有效性和性能提升。
Experiments
采用Qwen-3-235B-A22B、DeepSeek V3、Hy3模型，使用真实生产和合成数据集，比较传统全批次不变核与CoRun方案。指标包括输出一致性、吞吐率、时间-首字令牌、输出令牌时间。设置不同请求长度、批次大小，进行压力测试。通过消融实验验证隔离预填充、固定形状解码和随机数绑定的贡献。结果显示，CoRun在保证输出一致的同时，吞吐提升200%以上，延迟降低50%以上，验证其实用性。
Results
在多模型、多场景下，CoRun实现了完全一致的输出，显著优于传统方案。具体表现为：平均吞吐提升200%以上，时间-首字令牌减少51.8%，输出令牌时间缩短48.6%。在大模型（如Qwen-3-235B）上，延迟由88.4ms降至32.6ms，性能提升明显。消融实验显示，隔离预填充和请求绑定随机数是保证确定性的关键。系统在多请求并发环境中表现出优异的稳定性和效率，验证了其工程实用价值。
Applications
该技术适用于大规模模型的生产部署、模型调试、RL训练和模型评估场景。通过确保推理输出的可重复性，提升模型的可靠性和调试效率。尤其适合需要严格复现的科研和工业应用，如自动问答、内容生成、智能客服等。未来可结合多GPU调度和模型剪枝，进一步优化性能和能耗，推动大模型在实际场景中的广泛应用。
Limitations & Outlook
系统在极端长prompt或特殊算子场景下可能表现不佳，因请求形状控制难以覆盖所有情况。多GPU环境中，调度同步可能引入通信开销，影响扩展性。对硬件和软件版本依赖较强，存在迁移和兼容性挑战。未来需优化调度策略，增强系统的适应性和鲁棒性。
Plain Language Accessible to non-experts
想象你在厨房里做饭，每个厨师（核函数）都有自己的工作方式。有些厨师可以灵活调整烹饪顺序，但这样会导致菜肴味道不一致。为了保证每道菜都一样，厨房里规定厨师必须按照固定顺序操作（全批次不变核），但这样会让做饭变慢。本文提出一种新方法，就像让厨师在保持原有操作顺序的基础上，提前准备好所有食材（隔离预填充），并用统一的工具（固定形状解码）快速完成菜肴。这样既保证了菜的味道一致，又提高了做饭速度。这就像在模型推理中，控制请求的形状和顺序，确保每次输出都一样，同时提升效率。
ELI14 Explained like you're 14
想象你在学校的食堂点餐，每次点的菜都要排队等候。有时候，排队顺序会变，导致你拿到的菜不一样，味道也可能不同。为了让每次都吃到一样的菜，厨师可以提前准备好所有食材，按照固定的菜谱做饭，不管你什么时候来，都用同样的步骤。这样，不管排队顺序怎么变，你吃到的菜都一样，而且速度还快了很多。这就像这篇论文里的方法，控制模型请求的形状和顺序，让每次生成的内容都一样，还能更快完成任务。它就像让厨房变得更聪明、更快，保证每次都能吃到一样的美味！
Abstract
Despite fixed sampling parameters and random seeds, Large Language Model (LLM) inference exhibits output inconsistency, which undermines downstream tasks such as model evaluation and reinforcement learning. A major source of this nondeterminism is batch-dependent GPU execution: dynamic input shapes change kernel tiling and floating-point reduction orders. Existing systems address this problem with batch-invariant kernels, but these kernels restrict optimized tiling and split reductions, increasing more than 2$\times$ latency and reducing serving throughput by up to 74 %. This paper observes that although most kernels are not batch-invariant, they are position-invariant. Leveraging this property, we present CoRun, a scheduling-based system that achieves deterministic inference without requiring batch invariance. CoRun employs isolated prefill and fixed-shape batched decode to handle the two stages of LLM inference, respectively, leveraging CUDA graphs for efficient execution and simplified implementation. Experiments on LLMs with diverse architectures, including Qwen and DeepSeek, show that CoRun ensures determinism while improving throughput by 15-324 % over batch-invariant approaches, reducing time-to-first-token by 51.8 % and time-per-output-token by 48.6 % on average.