ResTok: Learning Hierarchical Residuals in 1D Visual Tokenizers for Autoregressive Image Generation
ResTok introduces hierarchical residual visual tokenizer, achieving gFID 2.34 with only 9 steps on ImageNet-256.
Key Findings
Methodology
ResTok builds hierarchical representations by progressively merging image and latent tokens within a Transformer framework. It introduces semantic residuals to reduce information overlap and enhance latent distribution concentration. The model combines multi-scale feature fusion with residual learning, resulting in richer representations. A hierarchical autoregressive generator predicts entire levels of latent tokens at once, drastically reducing sampling steps. The approach incorporates pre-trained vision foundation models for alignment, accelerating convergence. The entire pipeline involves encoding, residual merging, quantization, and hierarchical decoding, emphasizing the importance of residuals and multi-scale fusion for improved performance.
Key Results
- On ImageNet-256, ResTok achieves a gFID of 2.34 with only 9 sampling steps, outperforming previous AR models. It demonstrates a 3x efficiency gain over traditional single-hierarchy tokenizers. The model produces high-fidelity images with detailed semantics, verified through extensive ablation studies. Residual mechanisms significantly reduce codebook entropy, leading to more stable and concentrated latent distributions. The experiments confirm that hierarchical residual priors enhance both quality and speed of AR image generation.
Significance
This work addresses fundamental limitations of language-inspired visual tokenizers by emphasizing the hierarchical and residual properties inherent in vision. It advances the state-of-the-art in efficient, high-quality autoregressive image synthesis, bridging the gap between semantic richness and generation speed. The approach offers a new paradigm for multi-scale, controllable visual generation, with broad implications for AI-driven content creation, virtual reality, and multi-modal modeling, fostering more natural and controllable AI-generated visuals.
Technical Contribution
ResTok introduces a novel hierarchical residual framework that integrates multi-scale feature fusion and semantic residual learning into the visual tokenizer. It innovates with a level-wise autoregressive generator that predicts entire hierarchies, reducing sampling steps dramatically. The model leverages pre-trained vision foundation models for alignment, ensuring faster convergence and semantic consistency. These contributions collectively push the boundary of AR image generation, enabling high-quality synthesis with fewer steps and more meaningful latent representations.
Novelty
This is the first work to embed hierarchical residuals directly into 1D visual tokenization, combining multi-scale fusion with semantic residual learning. Unlike prior methods relying on handcrafted frequency or spatial hierarchies, ResTok learns residuals adaptively, capturing semantic attributes at different levels. The hierarchical autoregressive generator further distinguishes this approach by predicting entire levels simultaneously, significantly improving efficiency without sacrificing quality.
Limitations
- The model's training complexity and computational cost are high, requiring substantial resources. Future work should focus on reducing complexity for real-time applications. In extremely high-resolution scenarios, performance may degrade due to limited capacity. Dependence on pre-trained foundation models could limit generalization in unseen domains. Further research is needed to optimize residual learning and extend to videos or 3D data.
Future Work
Future directions include optimizing model efficiency for real-time deployment, exploring adaptive residual learning strategies, and extending hierarchical residual concepts to video and 3D scene synthesis. Integrating more powerful foundation models and multi-modal data will further enhance controllability and diversity of generated content. Additionally, developing lightweight architectures and unsupervised residual learning methods could broaden practical applications.
AI Executive Summary
ResTok introduces a hierarchical residual visual tokenizer that fundamentally rethinks how images are represented and generated. Traditional AR models, often based on language-inspired transformer architectures, treat visual data as flat sequences, neglecting the inherent hierarchical and residual nature of vision. ResTok addresses this by constructing multi-scale hierarchical representations through progressive merging of image and latent tokens, guided by semantic residuals that reduce redundancy and concentrate the latent distribution. This design enables the model to capture rich semantic information at different levels, facilitating more accurate and detailed image synthesis.
A key innovation is the hierarchical autoregressive generator, which predicts entire levels of latent tokens simultaneously rather than token-by-token. This approach drastically reduces sampling steps—down to just 9 in the case of ImageNet-256—while maintaining high image fidelity. Extensive experiments demonstrate that incorporating hierarchical residual priors significantly improves AR image generation, achieving a gFID of 2.34, outperforming existing methods in both quality and efficiency.
The significance of this work lies in its reconceptualization of visual tokenization, emphasizing the importance of hierarchical and residual properties that are fundamental to vision but often overlooked in language-based models. By integrating multi-scale features and semantic residuals, ResTok not only enhances representational capacity but also accelerates generation, paving the way for more controllable, high-quality, and efficient visual synthesis.
Despite its strengths, the model's complexity and reliance on pre-trained foundation models pose challenges for broader deployment. Future research will focus on optimizing computational costs, extending the framework to videos and 3D data, and exploring adaptive residual mechanisms. Overall, ResTok marks a significant step forward in the development of intelligent, scalable, and semantically rich image generation systems.
Deep Analysis
Background
近年来，Transformer在视觉任务中的应用不断扩大，尤其在图像生成领域，VQ-VAE等离散潜在编码技术推动了高效生成模型的发展。然而，现有的自回归标记器多沿用语言模型架构，忽视了视觉的层级和残差特性，导致潜在空间分散，影响生成质量。多尺度和层级结构的引入旨在增强模型的表达能力和生成效率。尽管如此，许多方法未能充分利用视觉的层级关系，存在潜在信息冗余和分布分散的问题。ResTok的出现，试图通过引入多尺度残差和语义残差机制，弥补这些不足，提升潜在表示的集中度和表达能力，从而实现更高效、更高质量的图像生成。
Core Problem
当前的1D视觉标记器多基于语言模型架构，缺乏对视觉层级和残差特性的充分利用，导致潜在分布分散，影响生成质量和效率。缺少跨层级特征融合机制，限制了模型的表达能力。同时，潜在代码本的高熵和信息重叠问题，使得模型难以在有限采样步骤内生成高质量图像。如何设计一种既能捕获多尺度信息，又能降低潜在熵的标记器，是当前面临的核心难题。
Innovation
ResTok的创新点包括：1）引入多尺度层级残差结构，通过逐步合并实现丰富的多尺度特征融合；2）学习语义残差，减少潜在空间信息重叠，提升分布集中度；3）设计层级自回归生成器，能在较少采样步骤内预测完整层级潜在标记；4）结合预训练视觉模型进行特征对齐，加快训练收敛。这些创新突破了传统单一层次表示的局限，为高效、细腻的图像生成提供了新思路。
Methodology
- �� 编码：用CNN提取图像特征，生成初始图像标记。• 多尺度融合：在Transformer中每N/S层插入残差合并块，将不同尺度的特征逐步融合，形成多尺度层级表示。• 语义残差：在潜在空间引入残差学习，减少信息重叠，提升潜在分布集中度。• 量化：用向量量化将潜在特征离散化，形成紧凑的潜在代码簿。• 解码：利用Transformer解码潜在标记，结合残差信息，重建高质量图像。• 生成：采用层级自回归生成器，逐层预测潜在标记，缩短采样步骤。• 对齐：引入预训练视觉基础模型进行特征对齐，加速训练。整个流程强调多尺度残差和层级特征的学习与融合。
Experiments
在ImageNet-256数据集上，ResTok训练200轮，采用128潜在标记，结合多尺度残差机制。对比多种基线模型，包括单层标记器和其他离散潜在模型。评估指标包括gFID、rFID、采样步数等。通过消融实验验证层级残差和多尺度融合的贡献。结果显示，ResTok在仅用9步采样下实现gFID 2.34，优于大部分对比方法，验证了模型的高效性和优越性能。还进行了潜在分布分析，发现残差机制显著降低了潜在熵。
Results
ResTok在ImageNet-256上实现gFID 2.34，采样步数仅为9，远优于传统方法如LlamaGen（gFID 3.07，576步）。潜在分布更集中，生成图像细节丰富、语义一致。引入层级残差机制显著提升了潜在空间的表达能力，减少信息冗余，增强模型稳定性。实验还验证了多尺度融合对提升生成质量的作用，显示出模型在多样性和细节表现上的优势。
Applications
ResTok适用于高质量图像生成、图像编辑、内容创作等场景。其高效的采样能力使其在实时应用中具有潜力，尤其适合需要快速生成高分辨率图像的行业。结合预训练模型，可实现更丰富的多模态内容生成，为虚拟现实、游戏设计等提供技术支持。
Limitations & Outlook
模型训练成本较高，依赖大量计算资源。在超高分辨率或极端复杂场景下表现仍有限。对预训练模型的依赖可能限制泛化能力。未来需优化模型结构，降低复杂度，提升在多样场景中的适应性。
Plain Language Accessible to non-experts
想象你在做一份大餐。传统厨师会按照食谱一步步操作，可能忽略了不同食材之间的关系。而ResTok就像一个聪明的厨师，他会先准备好不同层次的食材，比如基础调料、配料和装饰，然后根据菜肴的层次逐步组合。每一层都加入一些特殊的调味料（残差），让菜肴更丰富、更有层次感。这样做不仅让菜肴更美味，还能节省时间，因为他知道每一层的关系和作用。最终，这个厨师能在很短时间内做出一份色香味俱佳的菜肴。这就像ResTok用多层次的残差和融合，让图像生成变得更快、更好看。
ELI14 Explained like you're 14
你可以想象自己在拼一幅拼图。每次你先拼出大块的轮廓，然后再逐步填充细节。传统的方法就像一块一块拼，花很长时间。而ResTok像个聪明的朋友，他会先拼出大轮廓，然后用残差（补充的小细节）把图片变得更完整。每次拼完一层，就像在不断完善图片，让它变得越来越细腻。这样一来，你只需要少量的步骤，就能拼出一幅漂亮的画。这就像ResTok用多层次的拼接和残差，让图像生成既快又漂亮。
Glossary
Transformer（变换器）
一种深度学习模型，擅长处理序列数据，广泛用于自然语言和视觉任务。
ResTok的编码和解码核心架构。
潜在空间（Latent Space）
模型中用于表示压缩信息的抽象空间，便于生成和重建。
ResTok通过潜在空间进行图像编码和生成。
残差（Residual）
在神经网络中，用于补充或修正前一层输出的差异信息，增强模型表达能力。
ResTok引入多尺度残差以提升潜在表示。
gFID（条件Fréchet Inception Distance）
衡量生成图像与真实图像相似度的指标，数值越低越好。
用以评估ResTok生成图像的质量。
多尺度（Multi-scale）
在不同空间或语义层次上处理信息的方法，增强模型表达能力。
ResTok采用多尺度特征融合。
Open Questions Unanswered questions from this research
- 1 如何进一步降低模型复杂度以实现实时生成仍未解决。
- 2 多模态融合中的层级残差设计尚缺乏系统研究。
- 3 在超高分辨率场景下的潜在分布优化仍需探索。
Applications
Immediate Applications
高效图像生成
ResTok可用于快速生成高质量图像，适合内容创作、虚拟现实等行业，满足实时性和细节丰富的需求。
Long-term Vision
多模态内容创作
结合多模态信息，推动跨模态生成技术，未来实现更智能、更丰富的虚拟场景和交互体验。
Abstract
Existing 1D visual tokenizers for autoregressive (AR) generation largely follow the design principles of language modeling, as they are built directly upon transformers whose priors originate in language, yielding single-hierarchy latent tokens and treating visual data as flat sequential token streams. However, this language-like formulation overlooks key properties of vision, particularly the hierarchical and residual network designs that have long been essential for convergence and efficiency in visual models. To bring "vision" back to vision, we propose the Residual Tokenizer (ResTok), a 1D visual tokenizer that builds hierarchical residuals for both image tokens and latent tokens. The hierarchical representations obtained through progressively merging enable cross-level feature fusion at each layer, substantially enhancing representational capacity. Meanwhile, the semantic residuals between hierarchies prevent information overlap, yielding more concentrated latent distributions that are easier for AR modeling. Cross-level bindings consequently emerge without any explicit constraints. To accelerate the generation process, we further introduce a hierarchical AR generator that substantially reduces sampling steps by predicting an entire level of latent tokens at once rather than generating them strictly token-by-token. Extensive experiments demonstrate that restoring hierarchical residual priors in visual tokenization significantly improves AR image generation, achieving a gFID of 2.34 on ImageNet-256 with only 9 sampling steps. Code is available at https://github.com/Kwai-Kolors/ResTok.