SSD: Spatially Speculative Decoding Accelerates Autoregressive Image Generation

The evolution of image generation has seen significant advances with models like VQ-VAE, VQGAN, and MAGVIT-v2, which encode images into discrete tokens for autoregressive modeling. These approaches leverage transformer architectures to generate images token-by-token, enabling flexible and high-quality synthesis. However, as image resolutions increase, the quadratic growth of token sequences imposes a severe computational burden, known as the memory wall, limiting real-time applications. Existing acceleration techniques, such as multi-token prediction and speculative decoding, have achieved modest speedups (around 2-4×) but often at the cost of quality or require complex architectural modifications. Recent efforts to exploit spatial structure, like multi-row prediction, have shown promise but still face challenges in balancing speed and fidelity. The core issue remains: how to fully utilize the inherent 2D spatial relationships in images to break through the quadratic complexity barrier. This background underscores the importance of developing geometry-aware decoding strategies that can leverage the local correlations in images to enable faster and more efficient generation.

The fundamental problem addressed in this paper is the inefficiency of traditional autoregressive image generation models, which predict each pixel sequentially along a flattened 1D sequence. This leads to an inference complexity of O(n²) for an n×n image, resulting in prohibitively slow generation times, especially for high-resolution images. The bottleneck is compounded by the need to repeatedly load large model parameters for each token prediction, creating a memory bandwidth constraint. Existing acceleration methods, such as multi-token prediction and parallel decoding, either degrade image quality or require extensive architectural changes. Therefore, the key challenge is to design a decoding strategy that respects the intrinsic 2D spatial structure of images, enabling parallel prediction of pixel blocks while maintaining high fidelity and stability. Overcoming this bottleneck is crucial for enabling real-time, high-resolution visual synthesis in practical applications.

This paper introduces several innovative ideas: 1) The core innovation is the explicit modeling of 2D spatial correlations by predicting entire pixel blocks along both horizontal and vertical axes, rather than relying solely on linear sequence prediction. 2) The use of continuous latent space prediction, based on the last transformer layer's hidden states, improves stability and accuracy over discrete token prediction. 3) The multi-round auto-correcting verification mechanism allows the model to iteratively refine predictions, reducing errors and ensuring spatial coherence. 4) The modular design enables integration with existing pretrained models without architectural modifications, making the approach highly versatile. These innovations collectively enable the reduction of inference complexity from quadratic to linear, unlocking significant speedups while preserving high image quality.

• �� Starting from a pretrained autoregressive transformer, encode images into discrete tokens using a vector quantizer like VQ-VAE.
• �� Train lightweight horizontal and vertical prediction heads on the last transformer layer's continuous features, predicting the hidden states of neighboring pixels or pixel blocks.
• �� During inference, predict entire rows sequentially along the horizontal axis using the horizontal heads, then predict multiple subsequent rows in parallel along the vertical axis with the vertical heads, forming a 2D block prediction.
• �� Use continuous latent space prediction to enhance stability, with the predictor network trained via a smooth L1 loss against the ground-truth hidden states.
• �� Implement an auto-correcting verification process: after initial prediction, pass the predicted blocks through the backbone for validation, and iteratively refine predictions through multiple verification rounds, correcting minor errors.
• �� The process is modular, requiring no changes to the backbone, and can be applied to any discrete token-based autoregressive model.
• �� The entire pipeline emphasizes leveraging local spatial adjacency, reducing the total inference steps from O(n²) to O(n), enabling real-time high-resolution generation.

• �� The models evaluated include Janus-Pro-7B, Lumina-mGPT-7B, and Emu3-8B, tested on images of sizes 24×24, 48×48, and 90×90, respectively, covering a broad range of resolutions.
• �� Datasets used are public benchmarks DPG-Bench and GenEval, which assess semantic alignment and compositional fidelity.
• �� Baseline comparisons include standard autoregressive decoding, 1D multi-token prediction, and SJD, measuring inference time, speedup ratios, and image quality metrics such as FID and Inception scores.
• �� The lightweight prediction heads are trained on generated data using self-distillation, with datasets of 60,000, 20,000, and 5,000 samples for each model.
• �� Hyperparameters include 5 tokens per row, 1 verification round for horizontal prediction, and multi-row prediction with staged verification for larger grids.
• �� Ablation studies analyze the impact of prediction target space, number of verification rounds, and the effect of auto-correction, providing insights into the optimal configurations.

• �� The SSD method achieves up to 13.3× reduction in inference time across models, with the Emu3 model’s generation time dropping from 339 seconds to 25.55 seconds.
• �� The speedup is consistent across different resolutions and models, with larger images benefiting more from multi-row prediction.
• �� The quality of generated images remains high, with minimal degradation in metrics like FID, demonstrating that speed gains do not compromise fidelity.
• �� Ablation results show that predicting in the continuous latent space yields better draft accuracy than discrete token prediction, and multi-round auto-correction further enhances output quality.
• �� The experimental results validate the hypothesis that leveraging 2D spatial structure is key to unlocking massive computational efficiencies in visual autoregressive models.

• �� Immediate applications include real-time high-resolution image synthesis for virtual reality, gaming, and digital content creation, where speed and quality are critical.
• �� The method can be integrated into multimodal systems combining text and images, enabling faster and more coherent content generation.
• �� Long-term, SSD could facilitate the development of intelligent visual assistants, automated scene rendering, and interactive design tools, transforming industries such as entertainment, advertising, and education.

• �� The method relies on accurate training of lightweight prediction heads, which may struggle with highly complex or ambiguous scenes, leading to residual errors.
• �� The auto-correction process, while effective, introduces additional computation during verification rounds, which could impact real-time performance in resource-constrained environments.
• �� Current implementation focuses on discrete token models; extending to continuous pixel spaces or multimodal inputs remains an open challenge.
• �� The approach assumes the availability of sufficient training data for the spatial prediction heads, which might limit applicability in low-resource scenarios.
• �� Further research is needed to optimize the balance between speed, quality, and computational cost, especially for ultra-high-resolution generation.