VibeToken: Scaling 1D Image Tokenizers and Autoregressive Models for Dynamic Resolution Generations
VibeToken is a resolution-agnostic 1D Transformer image tokenizer enabling efficient AR image synthesis at arbitrary resolutions.
Key Findings
Methodology
VibeToken employs dynamic positional embeddings, variable patch sizes, and dynamic token length encoding, combined with pretraining strategies to achieve resolution-agnostic high compression. The core algorithms include Transformer-based sequence encoding, adaptive patch embedding, and multi-codebook quantization (MVQ), supporting 32-256 controllable tokens. VibeToken-Gen integrates class and resolution conditioning, creating a constant FLOPs generator. Trained on ImageNet-1k, it supports multi-resolution, multi-aspect ratios, significantly reducing inference computational costs.
Key Results
- VibeToken achieves 3.94 gFID on 1024×1024 images with only 64 tokens, outperforming diffusion models requiring 1024 tokens (5.87 gFID). Inference FLOPs are fixed at 179G, 63× more efficient than LlamaGen's 11T at 1024×1024.
- Supports arbitrary resolutions and aspect ratios with high quality, generating images faster (0.46s vs 1.08s) than NiT, while maintaining low gFID scores. Ablation studies confirm the effectiveness of dynamic positional embeddings and MVQ.
- Models demonstrate strong generalization across resolutions and ratios, maintaining high fidelity and efficiency, validated through extensive experiments on ImageNet-1k and FFHQ datasets.
Significance
This work addresses the core bottleneck of AR models' scalability by introducing a resolution-agnostic encoding scheme, enabling high-quality, real-time image generation across diverse scenarios. It bridges the gap between diffusion and AR models, offering a practical solution for industrial deployment. The approach paves the way for flexible, scalable, and computationally efficient visual synthesis, impacting fields from content creation to virtual reality. The innovation in resolution-independent encoding and constant FLOPs inference marks a significant step toward democratizing advanced generative models for widespread use.
Technical Contribution
The paper introduces VibeToken, a novel resolution-agnostic 1D Transformer tokenizer that combines dynamic position embeddings, adaptive patch sizes, and multi-codebook quantization, supporting flexible sequence lengths (32-256 tokens). Building on this, VibeToken-Gen is designed as a constant FLOPs autoregressive generator, capable of high-quality image synthesis at arbitrary resolutions. The integration of these components results in a scalable, efficient, and generalizable framework that surpasses existing fixed-resolution AR models in both compression and inference efficiency.
Novelty
This is the first work to propose a fully resolution-agnostic 1D Transformer tokenizer capable of encoding images of arbitrary resolution into a fixed, small token set, supported by dynamic position embeddings and multi-codebook quantization. Unlike prior fixed-resolution tokenizers, VibeToken maintains a short, controllable token sequence, enabling scalable AR generation. Its combination of resolution flexibility, compression efficiency, and constant FLOPs inference distinguishes it from existing methods, representing a significant leap in AR image modeling.
Limitations
- Despite impressive results, the model may still produce artifacts or detail loss in extremely non-standard aspect ratios or ultra-high resolutions, requiring further refinement.
- Training involves extensive multi-scale data and complex strategies, leading to high computational costs and potential overfitting risks.
- Current implementation focuses on 2D images; extending to video or 3D scenes involves additional challenges in temporal and spatial consistency.
Future Work
Future research will focus on enhancing multi-scale feature fusion, reducing training costs, and improving robustness in extreme scenarios. Exploring integration with other modalities like video and 3D data, as well as real-time deployment in industrial applications, are promising directions. Further, optimizing the multi-codebook quantization for even higher compression and fidelity will be prioritized.
AI Executive Summary
VibeToken introduces a transformative approach to image generation, addressing the longstanding challenge of scalability in autoregressive models. Traditional AR models, such as LlamaGen, suffer from a token count that grows quadratically with resolution, leading to prohibitive computational costs at high resolutions like 1024×1024. Diffusion models, while flexible, are computationally intensive and less suitable for real-time applications. To bridge this gap, the authors propose VibeToken, a resolution-agnostic 1D Transformer tokenizer that encodes images into a small, controllable sequence of tokens, supporting arbitrary resolutions and aspect ratios.
The core innovation lies in the dynamic position embedding, adaptive patch embedding, and multi-codebook quantization, which together enable the model to generalize beyond the training distribution. This design allows encoding images of any resolution into as few as 32 tokens, with the decoder capable of reconstructing images at any target resolution. The training strategy involves multi-scale, multi-aspect ratio sampling, ensuring robust generalization.
Building on this, VibeToken-Gen is a class-conditioned autoregressive generator that maintains a constant inference FLOPs of 179G regardless of resolution, enabling efficient high-resolution synthesis. Experimental results demonstrate that VibeToken-Gen produces 1024×1024 images with only 64 tokens and achieves a gFID of 3.94, outperforming diffusion models that require 1024 tokens and reach 5.87 gFID. The inference speed is also significantly improved, being 2.35 times faster than NiT at the same resolution.
This work significantly advances the scalability and efficiency of AR models, opening new possibilities for real-time, high-quality visual content creation across diverse applications. Despite current limitations in extreme aspect ratios and ultra-high resolutions, the proposed framework sets a new benchmark for resolution-flexible generative modeling, with promising directions for future research in multi-modal and multi-scale synthesis.
Deep Analysis
Background
图像生成技术经历了从GAN、VAE到扩散模型和AR模型的演变。扩散模型如DDPM在高质量生成方面表现优异，但计算成本高，难以实时部署。AR模型如LlamaGen、VQGAN+Transformer通过离散潜在空间实现高效生成，但在高分辨率和宽高比适应性方面受限。传统Tokenizers（如VQGAN）在Token数量随分辨率线性增长，导致推理成本剧增。近年来，1D Transformer Tokenizer（如TiTok、VAR）在压缩效率上取得突破，但仍受限于固定分辨率。解决方案亟需支持多尺度、多比例和超分辨率的编码机制，以满足工业应用需求。
Core Problem
现有AR模型在高分辨率场景中Token数量随分辨率线性增长，导致推理计算成本指数级上升，限制了其实际应用。传统Tokenizers难以支持宽高比变化和超分辨率重建，限制了模型的泛化能力。Diffusion模型虽能自然适应多尺度，但计算成本高，难以实现实时生成。如何在保证生成质量的同时，降低推理成本，成为核心难题。
Innovation
提出VibeToken，支持任意分辨率的动态位置嵌入和可变补丁机制，突破Token数量线性增长的瓶颈。结合多码本量化（MVQ）技术，实现极高压缩比和泛化能力。设计VibeToken-Gen，采用常数FLOPs推理策略，支持多尺度、多比例的高效生成。创新在于将Transformer序列编码与动态空间适应相结合，极大提升模型的灵活性和效率，推动AR模型工业化应用。
Methodology
- �� 设计动态位置嵌入，通过可调节的网格位置编码，支持不同分辨率和比例。
- �� 实现可变补丁大小，结合多尺度训练，增强模型泛化能力。
- �� 引入多码本量化（MVQ），提升压缩比，支持多尺度编码。
- �� 采用动态长度Token编码，支持从32到256 Token的可调节序列。
- �� 在ImageNet-1k上进行多尺度、多比例训练，优化模型的泛化能力。
- �� 构建VibeToken-Gen，结合类别和分辨率条件，设计常数FLOPs推理机制。
- �� 训练过程中采用随机采样分辨率和Token长度，增强模型的适应性。
Experiments
在ImageNet-1k、FFHQ等数据集上进行训练，比较VibeToken与传统Tokenizers（如LlamaGen、IBQ）在不同分辨率和比例下的重建质量（gFID）和推理效率。采用多尺度、多比例训练策略，验证模型的泛化能力。通过消融实验分析位置嵌入、补丁大小和Token长度对性能的影响，确保模型在高压缩比和多尺度场景下表现优异。
Results
VibeToken在1024×1024图像中仅用64 Token实现3.94 gFID，远优于需1024 Token的扩散模型（5.87 gFID）；推理FLOPs恒定在179G，显著优于传统AR模型（如LlamaGen在1024×1024时的11T FLOPs）。在多比例、多分辨率场景中，表现出优异的生成质量和泛化能力，支持任意宽高比和超分辨率重建。消融实验验证了动态位置嵌入和多码本量化的有效性，模型在不同场景下均保持高性能。
Applications
该技术可广泛应用于工业级图像生成、内容创作、虚拟现实和增强现实等场景，满足高效、多尺度、多比例的生成需求。支持实时内容生成和高质量超分辨率重建，推动智能内容生产的普及。未来还可结合视频和三维场景，拓展多模态生成应用。
Limitations & Outlook
模型在极端比例或超高分辨率场景中仍可能出现细节缺失或伪影，需进一步优化解码策略。训练成本较高，依赖大量多尺度、多比例样本，泛化能力受限于数据多样性。未来需结合更高效的编码和解码策略，提升模型鲁棒性和适应性。
Plain Language Accessible to non-experts
想象你在一家工厂里，工厂每天都要生产各种不同大小和形状的产品。过去，工厂的机器只能生产固定尺寸的产品，每次换尺寸都要重新调整设备，非常麻烦。而现在，这个工厂引入了一种新技术，可以根据订单的不同要求，自动调整生产线，快速生产出各种尺寸的产品。这就像VibeToken一样，它能把不同大小的图片编码成少量的“标签”，然后根据需要，快速“解码”出不同尺寸的图片。这样不仅节省了时间和成本，还能灵活应对各种需求。这种技术让图像生成变得像工厂生产一样高效、灵活，未来可以用在很多地方，比如游戏、电影、虚拟现实等。
ELI14 Explained like you're 14
想象你在玩一个超级酷的拼图游戏，你可以用很少的拼图片段拼出各种不同大小的图片。以前的拼图游戏每次都需要很多拼图片段，拼出大图片时特别慢。而这个新技术就像一个神奇的拼图工具，只用很少的拼图片段，就能拼出1024×1024的大图片，而且还能随意变大变小，就像魔法一样！它用一种聪明的方法，把图片的内容变成一串神奇的标签，然后再用这些标签拼出不同尺寸的图片。这样，不仅快，还能拼出各种奇形怪状的图片，非常厉害！未来，这个技术可以用在制作游戏、动画，甚至虚拟世界里，让我们的数字世界变得更加神奇和方便。
Glossary
Transformer（变换器）
一种基于自注意力机制的深度学习模型，擅长处理序列数据，广泛应用于自然语言和图像任务。
VibeToken利用Transformer编码图像序列，实现高效压缩和泛化。
多码本量化（MVQ）
一种将潜在空间划分为多个码本的量化技术，用于提升模型压缩比和表达能力。
MVQ在VibeToken中用于支持多尺度高效编码。
位置嵌入（Positional Embedding）
在Transformer中加入的位置信息，用于保持序列中元素的空间关系。
VibeToken采用动态位置嵌入以支持任意分辨率。
自回归模型（Autoregressive Model）
逐步预测序列中下一个元素的模型，广泛用于生成任务。
VibeToken-Gen是基于自回归机制的高效生成器。
gFID（生成质量指标）
衡量生成图像与真实图像差异的指标，数值越低代表质量越高。
实验中VibeToken实现了3.94的gFID，优于传统模型。
Open Questions Unanswered questions from this research
- 1 如何进一步提升极端比例和超高分辨率场景下的细节还原能力，仍是未来研究重点。
- 2 模型在多模态、多场景融合中的适应性和泛化能力尚未充分验证，需要更多跨领域实验。
- 3 优化训练成本和数据多样性，提升模型在实际工业环境中的鲁棒性和稳定性。
Applications
Immediate Applications
工业内容生成
支持高效、多尺度的图像内容生产，满足虚拟现实、游戏开发等行业的实时需求。
虚拟现实与增强现实
实现高质量超分辨率图像快速生成，提升沉浸体验和交互效果。
Long-term Vision
多模态内容创造
结合视频、三维场景等多模态数据，推动智能内容生产和虚拟世界的构建。
Abstract
We introduce an efficient, resolution-agnostic autoregressive (AR) image synthesis approach that generalizes to arbitrary resolutions and aspect ratios, narrowing the gap to diffusion models at scale. At its core is VibeToken, a novel resolution-agnostic 1D Transformer-based image tokenizer that encodes images into a dynamic, user-controllable sequence of 32-256 tokens, achieving a state-of-the-art efficiency and performance trade-off. Building on VibeToken, we present VibeToken-Gen, a class-conditioned AR generator with out-of-the-box support for arbitrary resolutions while requiring significantly fewer compute resources. Notably, VibeToken-Gen synthesizes 1024x1024 images using only 64 tokens and achieves 3.94 gFID; by comparison, a diffusion-based state-of-the-art alternative requires 1,024 tokens and attains 5.87 gFID. In contrast to fixed-resolution AR models such as LlamaGen -- whose inference FLOPs grow quadratically with resolution (11T FLOPs at 1024x1024) -- VibeToken-Gen maintains a constant 179G FLOPs (63.4x efficient) independent of resolution. We hope VibeToken can help unlock the wide adoption of AR visual generative models in production use cases.