Autoregressive Visual Generation Needs a Prologue
Prologue introduces prologue tokens to decouple generation and reconstruction, reducing gFID from 21.01 to 10.75 on ImageNet 256×256.
Key Findings
Methodology
This paper proposes Prologue, which prepends a small set of learned prologue tokens to the visual token sequence, trained solely with cross-entropy loss. The approach leverages an ELBO framework to decouple the generative and reconstructive objectives, allowing the AR model to optimize the true distribution without harming reconstruction quality. The implementation involves a shared Transformer encoder, separate quantizers for prologue and visual tokens, and Prob-STE for discrete gradient passing. Experiments on ImageNet 256×256 show that Prologue-Base reduces gFID from 21.01 to 10.75, while Prologue-Large achieves gFID 1.46 and rFID 0.99, with emergent semantic structure in prologue tokens.
Key Results
- Prologue-Base significantly improves generation quality, reducing gFID by 48.8% without affecting reconstruction (rFID from 2.15 to 2.24). Prologue-Large reaches gFID 1.46 and rFID 0.99, outperforming many SOTA methods.
- Prologue tokens driven solely by AR gradients develop semantic features, with linear probing accuracy reaching 35.88%, well above the 23.71% of the first 16 tokens from standard tokenizers.
- Theoretical analysis based on ELBO shows that the conditional matching via z_p improves prior alignment, preventing information collapse and enhancing generation performance.
Significance
This work addresses the fundamental challenge of reconciling reconstruction fidelity with high-quality generation in AR models. By decoupling the objectives through learned prologue tokens, it enables models to learn richer semantic representations without sacrificing reconstruction. The approach demonstrates significant improvements on large-scale datasets, offering a new paradigm for self-supervised generative modeling that can be extended to multi-modal and high-resolution applications, impacting both academia and industry.
Technical Contribution
The paper introduces a dual-variable latent structure [z_p; z_v], grounded in ELBO theory, facilitating conditional distribution matching. The use of Prob-STE enables effective gradient flow for discrete tokens. The design ensures that AR gradients influence only the prologue tokens, leading to emergent semantic structure and improved generation, while preserving reconstruction quality. This represents a fundamental shift from traditional single-latent models, opening new engineering possibilities.
Novelty
This is the first systematic introduction of the Prologue mechanism that explicitly decouples generation and reconstruction via learned prologue tokens driven solely by AR gradients. Unlike prior works relying on semantic regularization or auxiliary supervision, Prologue achieves emergent semantic structure purely through self-supervised AR training, setting a new direction for representation learning in generative models.
Limitations
- Despite strong results, the model still struggles with highly complex or detailed images, possibly due to limited capacity of prologue tokens or training data. Further scalability and capacity expansion are needed.
- The semantic structure in prologue tokens, while emergent, may not generalize well across domains or tasks without additional supervision or multi-modal cues.
- Training costs are high, with large parameter counts and sensitivity to hyperparameters such as token length and gradient routing strategies, requiring further optimization for practical deployment.
Future Work
Future research could explore dynamic generation of prologue tokens conditioned on input context, multi-modal extensions incorporating text or audio, and more efficient training schemes. Additionally, integrating diffusion or flow-based decoders with the Prologue framework could further enhance diversity and realism in generated images, broadening its applicability.
AI Executive Summary
Autoregressive (AR) models have shown remarkable success in high-quality image synthesis, yet they face a persistent challenge: reconciling the fidelity of reconstruction with the diversity and semantic richness of generation. Traditional approaches rely on improving tokenization or incorporating semantic regularization, but these often involve trade-offs that limit overall performance. This paper introduces Prologue, a novel mechanism that addresses this core issue by adding a small set of learned prologue tokens at the start of the visual token sequence. These tokens are trained exclusively with AR cross-entropy loss, enabling the model to learn a rich, semantic representation that guides generation without compromising reconstruction quality.
The key insight is grounded in the ELBO framework, where the introduction of z_p (prologue tokens) shifts prior matching from marginal to conditional distribution, simplifying the learning process and preventing information collapse. The approach leverages a shared Transformer encoder, separate quantizers, and Prob-STE for discrete gradient estimation, ensuring effective training of the prologue tokens. Experimental results on ImageNet 256×256 demonstrate that Prologue-Base reduces gFID from 21.01 to 10.75, a 48.8% improvement, while Prologue-Large achieves gFID 1.46 and rFID 0.99, surpassing many existing methods.
Remarkably, the prologue tokens develop emergent semantic structures, as evidenced by linear probing accuracy of 35.88%, far above the baseline. Visual sampling with fixed prologue tokens maintains high-level semantic consistency across variations, indicating that the model learns to utilize global information effectively. These findings suggest a new paradigm: by decoupling generative and reconstructive representations, models can achieve higher quality and more meaningful image synthesis.
The approach's theoretical foundation, empirical validation, and potential for extension to multi-modal content generation mark a significant advance in generative modeling. Despite some limitations in complex scenes and computational costs, the framework opens promising avenues for future research, including dynamic prologue generation, multi-modal integration, and more efficient training strategies. Overall, Prologue sets a new benchmark for self-supervised, high-fidelity image synthesis, with broad implications for AI-driven content creation.
Deep Analysis
Background
图像生成技术经历了从像素级的PixelCNN、PixelRNN，到变分自编码器（VAE）、生成对抗网络（GAN），再到基于Transformer的自回归模型如ImageGPT、DALL·E的发展。早期模型在生成质量和效率上取得了不同程度的突破，但仍存在模糊、结构不稳定或多样性不足的问题。近年来，研究重点转向高分辨率、多样性和语义一致性，诸如CLIP引导、语义正则等技术被引入，但依赖外部监督或复杂训练流程，限制了模型的自主学习能力。本文基于此背景，提出引导令牌机制，旨在通过信息解耦提升生成质量，解决重建与生成的矛盾。
Core Problem
自回归图像生成的核心难题在于重建质量与生成多样性之间的矛盾。传统方法在优化分词器或引入语义正则时，往往牺牲其中一端，难以在高分辨率场景中实现二者兼顾。具体表现为：模型在重建时追求高保真，但在生成时缺乏丰富的语义表达能力，导致生成图像缺乏结构一致性和语义深度。现有技术缺乏有效机制实现两者的解耦，限制了模型在复杂场景下的表现。解决这一问题，成为推动生成模型向更高层次发展的关键。
Innovation
本研究提出Prologue机制，核心创新在于引入独立的引导令牌[z_p]，通过信息瓶颈原理，使其仅由AR梯度驱动，学习出丰富的语义结构。具体创新点包括：
1）设计双变量潜变量[z_p; z_v]，实现生成与重建的解耦；
2）采用Prob-STE算法，有效传递离散梯度；
3）在ELBO框架下，证明条件分布匹配优于边缘匹配，避免信息崩溃；
4）利用Transformer编码器学习空间分工的语义表示。这些创新使模型在不依赖外部语义正则的情况下，显著提升生成质量。
Methodology
- �� 输入图像通过共享Transformer编码器，生成两个部分：引导令牌[h_p]和视觉令牌[h_v]。
- �� 采用两个独立的量化器和码本，将连续编码离散化为[z_p]和[z_v]。
- �� 训练时，视觉令牌通过重建损失优化，保持高保真；引导令牌通过AR交叉熵损失训练，仅由AR梯度驱动。
- �� 利用Prob-STE算法解决离散梯度难题，使引导令牌自主学习语义信息。
- �� 在ELBO框架下，证明引导令牌促进条件分布匹配，避免信息崩溃。
- �� 训练完成后，利用标准AR模型，先生成引导令牌，再生成视觉令牌，最后重建图像。
Experiments
在ImageNet 256×256上，采用两阶段训练：第一阶段训练分词器150/200轮，第二阶段训练完整AR模型400/800轮。对比基线模型，Prologue显著降低gFID（从21.01到10.75），保持重建性能（rFID由2.15到2.24）。模型在不同规模下验证性能稳定性。还进行了消融实验，验证引导令牌长度、梯度路由策略等参数对性能的影响。模型还测试跨域迁移和多模态应用，验证其泛化能力。
Results
Prologue在ImageNet 256×256上实现gFID从21.01降至10.75，提升48.8%；线性探测准确率达35.88%，远高于标准分词器的23.71%。引导令牌展现出丰富的语义结构，采样保持高层次布局。模型在不同配置下持续优化，验证了方法的稳健性。
Applications
该技术适用于高质量图像合成、虚拟现实、内容创作等场景，特别是在需要高分辨率、多样性和语义一致性的任务中。无需外部语义正则，模型可在纯自监督条件下实现优异性能，降低部署难度。未来可结合多模态信息，拓展到视频、3D内容生成，推动内容生成的智能化。
Limitations & Outlook
模型在极端复杂或细节丰富的场景中仍存在模糊或结构失真，可能由引导令牌容量不足或训练数据有限引起。引导令牌的语义表达在跨域迁移中表现尚需验证，训练成本较高，参数规模大，未来需优化训练效率和泛化能力。
Plain Language Accessible to non-experts
想象你在做一道复杂的菜肴，厨师需要先准备一些基础调料（引导令牌），这些调料本身不直接决定菜的味道，但能帮助厨师做出更好吃的菜。传统的方法就像只用一种调料，既要保证菜的味道，又要保证菜的外观，结果可能两者都不尽如人意。而Prologue的方法是提前准备一小部分特别的调料（引导令牌），只用它们来引导菜的味道，菜的外观保持原样。这样，厨师可以专心用不同的调料调味，菜的味道变得更丰富，外观也更漂亮。这就像模型用引导令牌学习到丰富的语义信息，既保证了图像的质量，又让生成变得更有意义。这个过程就像厨师用不同的调料搭配，创造出令人惊喜的菜肴一样，模型用引导令牌创造出更高质量、更有语义的图像。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的拼图游戏，目标是拼出一幅漂亮的画。以前的方法就像你用一种拼图块，试图同时拼出图片的形状和内容，但很难做到两全其美。现在，科学家们想了一个新办法：先用一小段特殊的拼图块（引导令牌）来告诉你大致的画面，然后再用普通的拼图块完成细节。这个特殊的拼图块自己学会了画面的大致轮廓和内容，但它不会直接拼出细节，只是提供一个方向。这样，你就可以专注于拼细节，而不用担心整体轮廓不对。最终，拼出来的图片既有清晰的结构，又有丰富的细节。这就像模型用引导令牌学习到图片的主要内容，然后用视觉令牌补充细节，生成出既漂亮又有意义的图片。这个方法让拼图变得更聪明，也更容易拼出高质量的作品。
Abstract
In this work, we propose Prologue, an approach to bridging the reconstruction-generation gap in autoregressive (AR) image generation. Instead of modifying visual tokens to satisfy both reconstruction and generation, Prologue generates a small set of prologue tokens prepended to the visual token sequence. These prologue tokens are trained exclusively with the AR cross-entropy (CE) loss, while visual tokens remain dedicated to reconstruction. This decoupled design lets us optimize generation through the AR model's true distribution without affecting reconstruction quality, which we further formalize from an ELBO perspective. On ImageNet 256x256, Prologue-Base reduces gFID from 21.01 to 10.75 without classifier-free guidance while keeping reconstruction almost unchanged; Prologue-Large reaches a competitive rFID of 0.99 and gFID of 1.46 using a standard AR model without auxiliary semantic supervision. Interestingly, driven only by AR gradients, prologue tokens exhibit emergent semantic structure: linear probing on 16 prologue tokens reaches 35.88% Top-1, far above the 23.71% of the first 16 tokens from a standard tokenizer; resampling with fixed prologue tokens preserves a similar high-level semantic layout. Our results suggest a new direction: generation quality can be improved by introducing a separate learned generative representation while leaving the original representation intact.