Blockwise Parallel Decoding for Deep Autoregressive Models

Deep autoregressive models, especially Transformer architectures, have set new standards in sequence generation tasks like machine translation, summarization, and image synthesis. Their success stems from powerful parallel training mechanisms, but inference remains bottlenecked by sequential token generation. Existing acceleration methods—such as knowledge distillation, non-autoregressive models, and subscaling—offer speedups but often compromise quality or require complex modifications. Transformer’s self-attention enables parallel scoring of all positions, which inspires new decoding strategies. Despite these advances, achieving high-speed, high-fidelity generation remains challenging, especially in real-time scenarios. This context motivates the development of efficient decoding algorithms that exploit model parallelism without sacrificing output quality.

The core challenge in autoregressive decoding is the inherently sequential nature of token generation, which limits inference speed. Existing solutions either reduce quality or demand significant architectural changes. For real-world applications like live translation or video synthesis, this latency is unacceptable. The problem is compounded by the need to balance speed with output fidelity, especially when models are deployed on resource-constrained hardware. Developing a decoding scheme that can leverage parallel computation while maintaining the same output as greedy decoding is crucial. The difficulty lies in designing verification mechanisms that ensure correctness without adding excessive computational overhead.

The paper introduces a blockwise parallel decoding strategy that predicts multiple tokens in parallel using auxiliary models trained alongside the base Transformer. These models generate candidate sequences, which are then validated against the primary scoring model to find the longest prefix consistent with greedy decoding. This process allows skipping multiple decoding steps, reducing total iterations. Additionally, the authors propose a combined scoring and prediction model with a multi-output layer, enabling joint prediction of multiple tokens with minimal extra computation. Approximate decoding strategies, such as top-k selection and distance thresholds, further enhance speed while controlling quality loss. The approach is simple to implement, compatible with existing models, and supported by theoretical guarantees, making it a practical solution for accelerating autoregressive inference.

• �� Train auxiliary models to predict k future tokens in parallel, using the same input context.
• �� During inference, generate k tokens simultaneously with these models.
• �� Score each predicted token using the base model to verify the longest prefix that aligns with greedy decoding.
• �� If verification confirms the prefix, extend the hypothesis with these tokens.
• �� Merge verification and prediction steps by a combined model, reducing model calls.
• �� Use approximate strategies (Top-k, distance metrics) to allow some deviation, balancing speed and quality.
• �� Implement multi-output layers in Transformer decoders to support joint predictions.
• �� Fine-tune or freeze the base model to optimize performance.
• �� Incorporate knowledge distillation to improve prediction consistency.
• �� Evaluate on machine translation and image super-resolution datasets, adjusting block size k and approximation parameters.
• �� Measure iteration reduction, BLEU scores, visual quality, and wall-clock speedup to validate effectiveness.

The authors evaluate their method on WMT2014 English-German translation and CelebA image super-resolution datasets. They compare standard greedy decoding with blockwise parallel decoding across various block sizes (k=1, 2, 4, 6, 8, 10). Metrics include BLEU scores, mean accepted block size, and wall-clock time. The models are trained with and without knowledge distillation, and with fine-tuning of parameters. Ablation studies analyze the impact of approximate decoding strategies and combined scoring models. Results show consistent iteration reductions, with BLEU scores dropping less than 1 point at optimal settings, and visual quality maintained. Wall-clock speedups reach up to 4x in translation and 7x in super-resolution, demonstrating practical efficiency gains.

Decoding iterations reduced from m to as low as m/7, with BLEU only decreasing by ~0.8 points, achieving scores around 26.58. In image tasks, average accepted block size reached 6.79, with speedups up to 7x. Knowledge distillation and fine-tuning further improved these metrics, with minimal quality loss. Approximate decoding strategies like top-k selection enhanced speed, and the method proved robust across tasks, models, and parameter settings. Wall-clock speedups of 3-4x were observed, confirming real-time applicability.

The proposed decoding scheme can be integrated into existing Transformer-based NLP and vision models, enabling real-time translation, captioning, and image synthesis. It requires minimal architectural modifications, mainly adding multi-output layers and training auxiliary predictors. The approach is suitable for deployment in latency-sensitive environments like live translation services, virtual assistants, and interactive content creation. Its flexibility allows adaptation to various tasks, making high-quality generation feasible in practical scenarios with limited computational resources.

The method relies heavily on the Transformer’s parallel scoring capability, limiting applicability to architectures lacking such features. Verification steps may become bottlenecks for very long sequences, reducing speed gains. Approximate decoding introduces potential quality degradation, especially with aggressive approximation parameters. Further research is needed to optimize verification efficiency, extend to other architectures, and handle diverse sequence lengths robustly.