Autoregressive Image Generation Needs Only a Few Lines of Cached Tokens
LineAR compresses KV cache for efficient autoregressive image generation, reducing ImageNet FID from 2.77 to 2.68.
Key Findings
Methodology
LineAR is a training-free progressive KV cache compression pipeline that exploits the characteristics of visual attention, managing the cache in a 2D view and progressively evicting less informative tokens to enable efficient autoregressive image generation.
Key Results
- On LlamaGen-XL, ImageNet FID improved from 2.77 to 2.68, COCO FID from 23.85 to 22.86, retaining only 1/6 of the KV cache.
- On Lumina-mGPT-768, DPG scores improved with just 1/8 KV cache retained.
- Achieved 67.61% memory reduction and 7.57x speedup on LlamaGen-XL.
Significance
This study significantly reduces memory requirements and computational costs for autoregressive image generation, while improving generation speed and quality. It addresses the memory bottleneck caused by caching all generated tokens during decoding, offering new insights for multimodal generation.
Technical Contribution
LineAR introduces a novel cache compression method by managing KV cache in a 2D view, significantly reducing memory usage and computational costs without retraining the model. This opens new possibilities for applying autoregressive models in high-resolution image generation.
Novelty
LineAR is the first to introduce progressive KV cache compression in autoregressive image generation, differing from previous static truncation strategies by dynamically evicting less informative tokens for more efficient cache management.
Limitations
- In extremely high-resolution image generation, a substantial cache space may still be required to maintain generation quality.
- The method relies on the characteristics of visual attention and may not be applicable to all types of autoregressive models.
Future Work
Future research could explore the application of LineAR in other multimodal generation tasks and further optimize its performance across different model architectures.
AI Executive Summary
Autoregressive image generation has gained attention for its scalability and generality, but existing methods suffer from severe memory bottlenecks due to the need to cache all generated visual tokens during decoding. LineAR manages the cache in a 2D view, progressively evicting less informative tokens to achieve efficient autoregressive image generation. Experiments show that LineAR significantly improves generation speed and quality across multiple models while reducing memory requirements. This method requires no retraining and is applicable to various autoregressive image generation tasks. Although substantial cache space may still be needed for extremely high-resolution image generation, LineAR offers new insights and technical pathways for multimodal generation. Future research could further optimize its performance and explore its application in other generation tasks.
Deep Analysis
Background
Autoregressive models have achieved great success in text generation and are gradually being extended to image generation. Existing methods quantize images into discrete tokens and decode them in a GPT-style manner. However, caching all tokens during generation leads to memory bottlenecks that limit practical applications.
Core Problem
Autoregressive image generation requires caching all generated visual tokens, leading to memory usage that grows linearly with sequence length. This not only increases storage demands but also significantly reduces generation speed, posing a barrier to practical deployment.
Innovation
LineAR introduces a progressive cache compression method by managing KV cache in a 2D view. Unlike traditional one-shot truncation strategies, LineAR dynamically evicts less informative tokens, significantly improving cache management efficiency.
Methodology
- �� Manage cache in a 2D view, generating images line by line.
- �� Maintain initial anchor and recent lines' tokens, progressively evict less informative tokens from the mid-region.
- �� Use inter-line attention to guide compression, ensuring generation quality is not compromised.
Experiments
Experiments were conducted on six autoregressive image generation models, covering different architectures and generation resolutions. Generation quality was evaluated using ImageNet and COCO datasets, measuring memory and speed improvements.
Results
On LlamaGen-XL, ImageNet FID improved from 2.77 to 2.68, COCO FID from 23.85 to 22.86, with 67.61% memory reduction and 7.57x speedup. On Lumina-mGPT-768, DPG scores improved with just 1/8 cache retained.
Applications
LineAR is suitable for scenarios requiring efficient image generation, such as real-time image synthesis and multimodal generation tasks. Its memory and speed advantages make it particularly beneficial in resource-constrained environments.
Limitations & Outlook
In extremely high-resolution image generation, cache requirements remain high. The method relies on the characteristics of visual attention and may not be applicable to all autoregressive models. Future improvements could optimize compression strategies and explore other generation tasks.
Plain Language Accessible to non-experts
Imagine a factory where workers need to complete products based on previous steps. Traditional methods require workers to remember every detail, leading to memory overload. LineAR acts like a smart factory manager, knowing which steps are crucial and which can be skipped. This way, the factory can produce high-quality products faster without remembering every detail.
ELI14 Explained like you're 14
Imagine you're playing a game that requires you to remember many steps. Traditional methods are like asking you to remember every single step, while LineAR is like a smart assistant that tells you which steps are important and which can be skipped. This way, you can finish the game faster without getting a headache from remembering too much!
Glossary
Autoregressive
A generation model that generates sequences step by step by predicting the next token.
In image generation, autoregressive models are used to generate each part of the image progressively.
KV Cache
Stores key-value pairs of all tokens generated during the process to avoid recalculating projections.
In autoregressive generation, KV cache stores generated visual tokens.
Visual Attention
A mechanism that focuses on specific areas of an image during generation.
LineAR leverages visual attention characteristics to optimize cache management.
FID (Fréchet Inception Distance)
A metric for evaluating the quality of generated images; lower values indicate better quality.
Used to evaluate LineAR's generation quality on ImageNet and COCO datasets.
DPG (Dataset for Prompted Generation)
A dataset for evaluating model performance in prompted generation tasks.
Used to evaluate LineAR's performance on Lumina-mGPT-768.
Open Questions Unanswered questions from this research
- 1 How can cache requirements be further reduced in extremely high-resolution image generation? Current methods still require substantial cache space at high resolutions.
- 2 How does LineAR perform in other multimodal generation tasks? Further experimental validation is needed.
Applications
Immediate Applications
Real-time Image Synthesis
LineAR can be used in applications requiring fast image generation, such as gaming and virtual reality, reducing memory needs and improving generation speed.
Long-term Vision
Multimodal Generation
LineAR's cache compression technique can be applied to multimodal generation tasks, driving the development of more efficient generation models.
Abstract
Autoregressive (AR) visual generation has emerged as a powerful paradigm for image and multimodal synthesis, owing to its scalability and generality. However, existing AR image generation suffers from severe memory bottlenecks due to the need to cache all previously generated visual tokens during decoding, leading to both high storage requirements and low throughput. In this paper, we introduce \textbf{LineAR}, a novel, training-free progressive key-value (KV) cache compression pipeline for autoregressive image generation. By fully exploiting the intrinsic characteristics of visual attention, LineAR manages the cache at the line level using a 2D view, preserving the visual dependency regions while progressively evicting less-informative tokens that are harmless for subsequent line generation, guided by inter-line attention. LineAR enables efficient autoregressive (AR) image generation by utilizing only a few lines of cache, achieving both memory savings and throughput speedup, while maintaining or even improving generation quality. Extensive experiments across six autoregressive image generation models, including class-conditional and text-to-image generation, validate its effectiveness and generality. LineAR improves ImageNet FID from 2.77 to 2.68 and COCO FID from 23.85 to 22.86 on LlamaGen-XL and Janus-Pro-1B, while retaining only 1/6 KV cache. It also improves DPG on Lumina-mGPT-768 with just 1/8 KV cache. Additionally, LineAR achieves significant memory and throughput gains, including up to 67.61% memory reduction and 7.57x speedup on LlamaGen-XL, and 39.66% memory reduction and 5.62x speedup on Janus-Pro-7B.