Retrofitting Linear Attention into Diffusion Language Models

Recent advances in diffusion models have revolutionized generative tasks, especially in text and image synthesis. Unlike autoregressive models like GPT, diffusion models generate data through iterative denoising, offering more flexible inference. Large-scale diffusion language models (dLLMs), such as LLaDA 2.1, have demonstrated impressive capabilities but face significant efficiency challenges during inference. The core issue lies in the attention mechanism: traditional softmax attention scales quadratically with sequence length, leading to high computational and memory costs, especially in bidirectional settings. Prior efforts like LoLCAT and Hedgehog introduced linear attention mechanisms to reduce complexity in autoregressive models, but their application to diffusion models remains unexplored. The need for efficient, scalable inference in dLLMs motivates this research.

The main challenge is that current dLLMs rely on full-sequence attention during denoising, which becomes prohibitively expensive as sequence length grows. Although blockwise semi-autoregressive decoding mitigates some costs, each denoising step still attends to all previous blocks, incurring prefix-attention costs. This limits real-time deployment and scalability, especially under high concurrency. Existing linear attention methods cannot be directly applied due to the bidirectional nature of diffusion models, which require maintaining full context for accurate denoising. Therefore, a new attention mechanism that balances efficiency and bidirectional interaction is needed to accelerate inference without degrading performance.

The paper introduces block-hybrid attention, which separates intra-block softmax attention from inter-block linear attention. Key innovations include: 1) within each active block, exact softmax attention ensures precise local interactions; 2) across blocks, the model compresses historical information into fixed-size recurrent states using Hedgehog kernels, enabling linear attention that is independent of sequence length; 3) a two-stage fine-tuning process aligns the hybrid attention with the original model, first via attention transfer, then via diffusion fine-tuning with LoRA. This design maintains bidirectional capabilities while significantly reducing computational complexity, enabling faster inference and higher concurrency.

• �� Design block-hybrid attention combining within-block softmax and cross-block linear attention via Hedgehog kernels. • During inference, compute exact softmax attention within the current block, ensuring local detail preservation. • Compress previous blocks’ keys and values into fixed-size recurrent states (Sn, Zn) using learned feature maps, independent of sequence length. • Implement a two-stage fine-tuning: first, freeze the backbone and optimize linear attention parameters using MSE loss against original attention outputs; second, attach LoRA adapters and perform diffusion training to recover performance. • Use a custom Triton kernel for efficient serving, integrating the fixed recurrent states into the attention computation pipeline. • Fine-tune only a small subset of parameters, ensuring low training cost and quick deployment.

• �� Conduct experiments on LLaDA 2.1-mini, a 16B parameter diffusion model, replacing 6 layers at indices {0,4,8,12,16,18}. • Use Tulu SFT dataset for training, with approximately 17 million tokens over 60 hours. • Evaluate on benchmarks including HumanEval, CMATH, MBPP, measuring accuracy and inference speed. • Compare the original and hybrid models, focusing on performance retention and throughput gains. • Test different concurrency levels, demonstrating up to 1.7× speedup at 128 requests, with throughput reaching nearly 4000 tokens/sec. • Ablation studies confirm the effectiveness of the layer selection and fine-tuning process.

• �� The hybrid model maintains high accuracy, with only a 3.6% drop on HumanEval and 1.6% on CMATH, while improving MBPP+ accuracy from 57.7% to 63.0%. • Inference throughput on a single GPU increases from 2310 to 3994 tokens/sec at 128 concurrent requests, a 1.73× speedup. • The entire retrofit process takes about 60 hours, demonstrating low cost and high efficiency. • The approach supports more concurrent requests under fixed memory, significantly improving deployment scalability.

• �� Enables real-time AI services like coding assistants, chatbots, and scientific reasoning tools by providing faster inference. • Suitable for deployment in resource-constrained environments, reducing hardware costs. • Facilitates scaling large diffusion models for industrial applications, including content creation, automated coding, and interactive AI systems. Long-term, this method can be integrated into model compression pipelines, enabling broader accessibility and deployment of large models in various domains.

• �� Only partial layer linearization limits the maximum speedup; full-layer linearization may cause larger performance degradation. • The method’s effectiveness on longer sequences and more complex tasks remains to be validated. • Fixed-size cross-block states may restrict information retention over very long contexts, affecting accuracy in long-form generation. Future work should explore adaptive layer selection, longer sequence handling, and multi-GPU scaling to overcome these limitations.