Improving Flexible Image Tokenizers for Autoregressive Image Generation
ReToK integrates redundant token padding and hierarchical semantic regularization, significantly enhancing autoregressive image generation for long sequences.
Key Findings
Methodology
ReToK combines redundant token padding with hierarchical semantic regularization. Additional tokens are appended to the sequence tail during training to activate tail tokens more frequently, alleviating information concentration in early tokens. Semantic regularization aligns early token features with high-level semantic features from pre-trained models like DINOv2, gradually reducing regularization strength to enable detailed low-level reconstruction. The training employs multi-scale reconstruction losses, including pixel, perceptual, and adversarial components, to improve long-sequence utilization. Extensive experiments on ImageNet 256×256 demonstrate superior performance across token lengths (32-256), outperforming existing flexible and fixed tokenizers.
Key Results
- ReToK achieves a gFID of 2.66 at 256 tokens, outperforming baseline flexible tokenizers like One-D-Piece (2.75) and fixed models like GigaTok (2.86), indicating enhanced long-sequence generation quality.
- Redundant token padding activates tail tokens, leading to more balanced information distribution and richer detail in generated images.
- Hierarchical semantic regularization improves early token semantic representations, reducing blurriness and enhancing detail fidelity, especially at short token lengths (32, 64).
Significance
This work addresses the fundamental bottleneck in flexible Tokenizers for long-sequence image generation, enabling models to better utilize all tokens and produce higher quality, more detailed images. It advances the state-of-the-art in scalable, flexible image tokenization, with broad implications for high-resolution content creation, multi-modal integration, and real-world applications such as virtual reality and content synthesis. The proposed techniques open new avenues for research in efficient long-sequence modeling and semantic-rich token representations, promising to elevate the capabilities of generative AI in both academia and industry.
Technical Contribution
ReToK introduces a novel redundant token padding mechanism that encourages tail token activation, coupled with hierarchical semantic regularization that aligns early token features with high-level semantics from pre-trained vision models. This combination effectively redistributes information across longer sequences, enabling better utilization of tail tokens. The approach employs multi-scale losses and decoder fine-tuning to optimize both semantic and low-level details. These innovations differ from prior nested dropout-based methods by actively promoting tail token engagement and semantic consistency, resulting in improved long-sequence generation stability and detail fidelity.
Novelty
This is the first systematic integration of redundant token padding with hierarchical semantic regularization to solve the long-sequence bottleneck in flexible image tokenizers. Unlike previous methods that suffer from information over-concentration in early tokens, ReToK actively activates tail tokens and aligns early semantic features with high-level representations, providing a new paradigm for scalable, flexible image tokenization. Its dual mechanism addresses core limitations of existing approaches, marking a significant step forward in the field.
Limitations
- Despite improvements, the method still faces challenges in extremely long sequences (e.g., 512+ tokens), where information sparsity and detail loss become prominent, requiring further optimization.
- Additional token padding increases computational complexity and training time, potentially limiting scalability and deployment efficiency.
- Dependence on pre-trained vision models like DINOv2 may restrict adaptability across diverse datasets or modalities without retraining or fine-tuning.
Future Work
Future research could focus on developing adaptive regularization strategies that dynamically balance semantic alignment and detail preservation across varying sequence lengths. Exploring multi-modal extensions, such as integrating text or audio cues, could further enrich token representations. Additionally, optimizing computational efficiency through model compression and inference acceleration will be crucial for real-world deployment. Extending the framework to handle ultra-long sequences (e.g., 512-1024 tokens) and multi-dimensional content like videos or 3D models remains an open and promising avenue.
AI Executive Summary
The rapid progress in deep generative models has revolutionized image synthesis, yet current tokenization strategies face significant limitations in balancing flexibility and detail preservation. Fixed-length tokenizers, while effective, lack adaptability to diverse image complexities, prompting the development of flexible, variable-length tokenization methods. However, existing approaches such as FlexTok and One-D-Piece encounter a critical bottleneck: as the sequence length increases, the quality of generated images stagnates or degrades, primarily due to information over-concentration in early tokens. This challenge hampers the potential of autoregressive models to produce high-fidelity, detailed images at longer sequences.
In response, the authors propose ReToK, a novel flexible tokenizer that integrates redundant token padding with hierarchical semantic regularization. The core idea is to append extra tokens at the sequence tail during training, which, combined with nested dropout, encourages the activation of tail tokens and redistributes information more evenly across the sequence. Simultaneously, by aligning early token features with high-level semantic representations from pre-trained vision models like DINOv2, ReToK enhances the semantic richness of early tokens, enabling better long-range information utilization.
Extensive experiments on ImageNet 256×256 demonstrate that ReToK outperforms existing flexible tokenizers, achieving a gFID of 2.66 at 256 tokens, surpassing baseline methods. The combination of tail token activation and semantic regularization significantly improves both image quality and reconstruction fidelity, especially at shorter token lengths. The approach not only bridges the gap between flexible and fixed-length tokenizers but also opens new avenues for scalable, high-quality image synthesis.
While promising, the method still faces challenges in ultra-long sequences and computational costs. Future work will explore adaptive regularization, multi-modal extensions, and efficiency improvements. Overall, ReToK marks a substantial step forward in flexible image tokenization, promising broader applications in high-resolution content creation, multi-modal AI, and beyond.
Deep Analysis
Background
图像生成技术经历了从像素重建到深度学习驱动的语义理解的演变。早期方法如VQ-VAE和VQ-GAN通过离散Token实现了图像压缩与生成，推动了生成模型的快速发展。近年来，Transformer架构的引入带来了更强的表达能力，诸如TiTok、GigaTok等模型实现了更高效的Token编码。然而，这些模型多采用固定长度或二维网格结构，难以捕获自然图像的复杂多样性，限制了模型的灵活性和细节还原能力。灵活长度的Tokenizer如FlexTok试图突破这一限制，但在长序列生成中表现不佳，信息过度集中在前部Token，导致生成质量随着Token数增加而下降。为解决这一问题，本文提出结合冗余Token填充和层级语义正则化的创新策略，旨在改善长序列中的信息分布和细节还原能力。
Core Problem
核心问题在于现有灵活Tokenizer在长序列生成中存在信息利用不足的瓶颈。传统嵌套Dropout训练策略使得大部分图像信息集中在前部Token，尾部Token几乎未被激活，导致模型难以充分利用尾部信息，限制了长序列的生成效果。随着Token数量的增加，生成质量反而下降，严重影响模型的实用性。此外，缺乏有效的尾部Token激活机制，使得模型在细节还原和语义表达方面表现不足，尤其在高分辨率、多样化场景中表现不佳。这些问题的根源在于信息分布不均和缺乏激活尾部Token的机制，亟需创新策略以改善信息利用效率。
Innovation
本文的创新点主要包括：1）引入冗余Token填充机制，在训练中添加尾部Token，激活尾部信息，改善信息分布不均；2）设计层级语义正则化，利用预训练视觉模型对早期Token进行高层次语义对齐，增强其表达能力，同时逐步放宽正则化，确保细节还原；3）结合多尺度重建损失，包括像素、感知和对抗损失，提升模型对长序列的利用效率和细节还原能力。这些创新突破了传统嵌套Dropout的局限，显著改善了长序列生成的效果，为灵活Tokenization提供了新的解决方案。
Methodology
- �� 采用GigaTok架构，结合Transformer编码器和解码器实现图像的离散Token化。• 在训练中引入冗余Token填充，将额外的尾部Token拼接到原始序列后，形成扩展序列。• 对扩展序列应用嵌套Dropout，激活尾部Token，促进信息均衡分布。• 利用预训练的DINOv2模型对早期Token的解码特征进行语义对齐，增强其高层次表达能力。• 逐步降低正则化强度，允许尾部Token捕获低层细节。• 结合多尺度重建损失，包括像素级、感知级和对抗级，提升整体重建质量。• 训练过程中采用多尺度正则化和解码微调，优化长序列生成效果。
Experiments
- �� 在ImageNet 256×256数据集上进行训练，采用不同Token长度（32-256）进行评估。• 比较基线GigaTok与改进版本ReToK的性能，指标包括gFID、rFID、PSNR、SSIM。• 设计消融实验验证冗余Token填充、层级语义正则化和解码微调的贡献。• 使用不同的自回归模型（LlamaGen系列）进行生成质量测试，确保结果的稳健性。• 通过多次实验调优正则化参数和训练策略，确保模型在长序列中的表现。
Results
- �� ReToK在256Token长度下实现的gFID为2.66，优于传统灵活Tokenizer（如One-D-Piece的2.75）和固定模型（如GigaTok的2.86），显示出长序列生成的优势。• 引入冗余Token填充后，尾部Token激活频率提升，贡献度分布趋于均衡，增强细节丰富度。• 层级语义正则化显著改善早期Token的语义表达能力，提升短Token（如32、64）生成的图像质量，减少模糊和失真。• 解码微调进一步优化了短序列的重建效果，整体性能达到或超过现有最优模型。
Applications
- �� 适用于高分辨率图像生成、内容创作、虚拟现实等场景，特别在需要长序列细粒度控制的应用中表现优越。• 结合预训练模型和灵活Tokenization，可实现多模态内容的高效编码与生成。• 未来可扩展至视频、3D模型等多维内容的生成与理解，推动智能内容创作产业升级。
Limitations & Outlook
- �� 在超长序列（如512以上Token）场景中仍存在信息稀疏和细节丢失的问题，需进一步优化正则化策略。• 训练引入冗余Token增加了计算负担，影响模型的训练效率和部署成本。• 依赖预训练视觉模型，迁移到不同任务或数据集时可能需要重新调优。
Abstract
Flexible image tokenizers aim to represent an image using an ordered 1D variable-length token sequence. This flexible tokenization is typically achieved through nested dropout, where a portion of trailing tokens is randomly truncated during training, and the image is reconstructed using the remaining preceding sequence. However, this tail-truncation strategy inherently concentrates the image information in the early tokens, limiting the effectiveness of downstream AutoRegressive (AR) image generation as the token length increases. To overcome these limitations, we propose \textbf{ReToK}, a flexible tokenizer with \underline{Re}dundant \underline{Tok}en Padding and Hierarchical Semantic Regularization, designed to fully exploit all tokens for enhanced latent modeling. Specifically, we introduce \textbf{Redundant Token Padding} to activate tail tokens more frequently, thereby alleviating information over-concentration in the early tokens. In addition, we apply \textbf{Hierarchical Semantic Regularization} to align the decoding features of earlier tokens with those from a pre-trained vision foundation model, while progressively reducing the regularization strength toward the tail to allow finer low-level detail reconstruction. Extensive experiments demonstrate the effectiveness of ReTok: on ImageNet 256$\times$256, our method achieves superior generation performance compared with both flexible and fixed-length tokenizers. Code will be available at: \href{https://github.com/zfu006/ReTok}{https://github.com/zfu006/ReTok}