Q-First: Most of Attention Needs Only the Query in Disaggregated LLM Decoding
Q-First protocol enables parallel attention decoding using only the query, achieving a relative error of 3.2×10^-3 without new operators or hardware.
Key Findings
Methodology
This paper introduces a protocol-based decoding scheme that prefetches only the query (Q) for attention, allowing the cache scan to proceed concurrently with feed-forward operations. Implemented on standard attention kernels without custom operators, the approach fine-tunes only Wq to keep errors below 3.2×10^-3. The architecture separates storage (KV cache) and compute (projection and feed-forward), enabling simultaneous execution of both devices. Validation on trained Qwen3-0.6B and 4B models shows errors below 0.0011 bits per byte, even with complex sub-layer exchanges. The protocol ensures minimal hardware changes, maintaining model parameters and shapes, while significantly improving decoding efficiency.
Key Results
- End-to-end validation yields an error of 4.2×10^-3, with no new operators or hardware, and only 9.5% of Wq fine-tuned. The approach handles sub-layer exchanges 25 times larger than traditional methods.
- Across multiple seeds, the maximum change in read points is 0.0011 bits/byte, well below the 0.0117 training resolution threshold, indicating negligible training disturbance.
- The architecture allows maximum overlap of attention and feed-forward, with two devices running concurrently, leading to substantial speedups and resource efficiency.
Significance
This work addresses the fundamental bottleneck in large-scale language model decoding—the serial dependency of attention and feed-forward layers. By enabling only query access for attention, it unlocks parallelism across hardware, reducing latency and resource consumption. The protocol’s simplicity and compatibility with existing kernels make it practical for deployment, paving the way for more efficient, scalable, and hardware-friendly large models. It also opens avenues for multi-device collaboration and model pruning, potentially transforming inference architectures.
Technical Contribution
The core innovation is a protocol that reorders attention computations, allowing the query to be read before the cache is fully populated. This is achieved without new operators or hardware modifications, relying solely on fine-tuning Wq. The architecture splits the model into storage and compute devices, with the latter responsible for projections and feed-forward, while the former handles KV cache sweeps. The method guarantees minimal error, verified through rigorous end-to-end testing, and supports cache splitting and merging, enhancing system flexibility.
Novelty
This is the first protocol to enable attention decoding using only the query, breaking the traditional serial dependence. By reordering sub-layers and designing a minimal interface for the cache device, it achieves near-perfect overlap without new operators. Unlike prior model or cache splitting methods, this approach maintains model integrity and introduces a novel, hardware-agnostic parallelization scheme that significantly improves decoding throughput.
Limitations
- The protocol requires multiple devices; single-device environments cannot leverage the parallelism, limiting its applicability in resource-constrained settings.
- Fine-tuning Wq introduces slight deviations, which, although minimal, may affect some tasks in extreme cases.
- Validation is primarily on specific models and datasets; broader generalization to other architectures and tasks remains to be demonstrated.
Future Work
Future efforts will focus on extending the protocol to larger models and diverse tasks, optimizing multi-device scheduling, and integrating hardware accelerators for even faster decoding. Exploring dynamic cache management and adaptive scheduling could further reduce latency. Additionally, research into reducing hardware dependencies will help democratize this approach for broader deployment.
AI Executive Summary
Large-scale language models (LLMs) have revolutionized natural language processing but face a critical bottleneck in decoding: the serial dependency of attention and feed-forward layers. Traditional methods require waiting for the attention output before proceeding, leading to underutilized hardware and increased latency. This paper introduces the Q-First protocol, a novel approach that enables attention to be computed concurrently with feed-forward operations by only requiring the query (Q) in the cache device. The key insight is that, at each decoding step, all but one of the attention score components depend solely on the query, allowing the cache device to scan the KV cache while the compute device continues its work.
The protocol is implemented on standard attention kernels without any custom operators or hardware modifications. Validation on trained Qwen3-0.6B and 4B models demonstrates an end-to-end error of just 4.2×10^-3, with only a small fine-tuning of Wq (9.5% of model weights). The approach maintains model shape and parameters, ensuring compatibility with existing frameworks. Experimental results show that moving the query read point earlier does not significantly impact model quality, with deviations well within training resolution thresholds.
This method effectively splits the decoding process across two hardware devices—one handling storage (KV cache) and the other computation (projection and feed-forward)—which run concurrently. Such a setup allows for near-perfect overlap, increasing decoding throughput by orders of magnitude, and handling complex sub-layer exchanges efficiently. The approach’s simplicity, accuracy, and hardware-agnostic nature make it a promising solution for deploying large models in resource-constrained or high-performance environments.
While requiring multiple devices, the protocol’s minimal error and compatibility with existing kernels suggest broad applicability. Future work will explore scaling to larger models, optimizing multi-device scheduling, and integrating hardware accelerators, aiming to further reduce latency and energy consumption. Overall, Q-First represents a significant step toward faster, more efficient large language model inference, with potential impacts across AI research and industry deployment.
Deep Dive
Abstract
Disaggregated LLM serving puts the KV-cache sweep on memory-optimised hardware and the projections and feed-forward on compute-optimised hardware, then inherits from the decoder block a dependency neither device wants: attention runs first and the feed-forward consumes its output, so within one sequence each side idles while the other works. The usual repair costs one resident KV cache per extra sequence in flight, which is what motivated separating the devices at all. We remove the dependency instead. The sweep needs only the query, and exchanging the two sub-layers makes that query available while the compute side still has work to do, so the two run concurrently; the current key and value follow as a cache write nothing waits on. We state the decode as a protocol, show that it runs on stock kernels, and verify it end to end on a trained checkpoint to a relative error of 3.2x10^-3 -- with no new operator, no changed shape and no new hardware. We then train the block 8 ways at two seeds each, varying only where the attention reads and holding everything else fixed. At three per cent of compute-optimal a lead in bits per byte measures how much a change disturbed training rather than what it reaches, so we read magnitudes and not rankings. Among the 5 blocks whose feed-forward does not consume their own attention, no read point differs from the one that moves nothing by more than 0.0026 bits per byte -- smaller than the gap between an arm and itself at a second seed, 0.0066 -- while the same runs resolve a sub-layer exchange 25 times as large. Moving the query early is a change the measurement cannot find, which is what the protocol needs. The reach is bounded: projecting every layer's query from the network's input costs +0.0974, refuting a pre-registered threshold at both seeds, so a query may be read one feed-forward early and no further back.