Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding

Autoregressive models like PixelCNN, PixelSNAIL, and transformer-based architectures such as DALL-E and Lumina-mGPT have revolutionized image synthesis, enabling high-fidelity, diverse outputs. These models operate by predicting tokens sequentially, which, while effective, leads to long inference times, especially for high-resolution images. Efforts to accelerate diffusion models via distillation and pruning have been successful, but similar techniques for autoregressive models are limited. Early attempts like Jacobi decoding enabled parallel token prediction but lacked support for sampling diversity, restricting their use in modern applications. As models grow in size and complexity, inference speed remains a bottleneck, hindering deployment in real-time scenarios.

The core challenge is reducing the inference latency of large autoregressive text-to-image models without sacrificing the diversity and quality of generated images. Traditional decoding methods are inherently sequential, making them slow for high-resolution outputs. While parallel decoding algorithms like Jacobi decoding exist, their deterministic convergence criteria are incompatible with sampling strategies that introduce randomness, essential for visual richness. This incompatibility prevents leveraging the full potential of sampling-based generation, creating a need for a new decoding paradigm that supports both speed and diversity.

The key innovation is the development of a probabilistic version of Jacobi decoding—SJD—that supports multi-token predictions in a single step. It employs a probabilistic acceptance criterion based on the ratio of conditional probabilities, allowing the model to accept multiple tokens simultaneously while maintaining the distribution fidelity. The method integrates a spatial locality-based initialization strategy, which uses neighboring token information to initialize new tokens more effectively, reducing the number of iterations needed. Unlike existing methods, SJD does not require additional training or auxiliary models, making it highly practical. The approach guarantees that accepted tokens follow the model's distribution, ensuring high-quality outputs.

• �� Start with pre-trained autoregressive transformer models and initialize the token sequence. • In each iteration, predict multiple tokens within a sliding window using a single forward pass. • Calculate the ratio of conditional probabilities for each token to determine acceptance. • Accept tokens probabilistically based on this ratio, fixing them for subsequent steps. • For rejected tokens, resample using a calibrated distribution informed by neighboring tokens. • Use spatial locality strategies to initialize new tokens, leveraging image structure. • Continue iterative decoding until convergence, with theoretical guarantees that accepted tokens follow the model distribution. • Employ a sliding window to balance memory and speed, enabling large-scale model inference.

Experiments conducted on Lumina-mGPT and Anole models used MSCOCO2017 for validation. Metrics included inference time, FID, and CLIP scores. The results showed that SJD reduced inference steps from over 2000 to fewer than 50, achieving approximately 2× acceleration with negligible quality loss. Ablation studies confirmed the effectiveness of spatial locality initialization. The method maintained high diversity and fidelity across different sampling strategies, demonstrating robustness. Hyperparameters such as window size and acceptance thresholds were tuned to optimize performance, with consistent improvements observed across scenarios.

SJD reduced Lumina-mGPT inference time from 87 seconds to about 42.7 seconds, doubling the speed with minimal FID and CLIP score variations. Similar results were observed with Anole, with speedups exceeding 2×. The spatial locality initialization further improved acceleration, reaching beyond 3× in simpler scenes. The generated images retained high quality and diversity, validating the method’s effectiveness across models and sampling methods. These results highlight the potential for practical deployment in real-time applications.

This method is suitable for real-time content creation, virtual reality, and interactive design, where fast, diverse image generation is critical. It can be directly applied to existing pre-trained models without retraining, facilitating widespread adoption. Future integration with hardware accelerators and adaptive initialization strategies could further enhance efficiency, enabling large-scale deployment in industry settings.

The approach assumes strong spatial locality, which may weaken in highly complex or abstract scenes. Randomness can introduce instability, especially in edge cases. Validation on extremely large models remains limited, and further work is needed to ensure robustness and stability in diverse scenarios. Future research should focus on adaptive strategies and hardware optimization to address these limitations.