Balancing Image Compression and Generation with Bootstrapped Tokenization
SelfBootTok decomposes images into global and local tokens via self-supervised learning, achieving 1.56 gFID with only 64 tokens, surpassing previous methods.
Key Findings
Methodology
SelfBootTok introduces a hierarchical framework that separates global semantic tokens from local detail tokens. It employs a multi-codebook design with differentiable soft vector quantization, enabling end-to-end training. The core innovation is a self-supervised paradigm where global tokens predict local details using a combination of MLP and autoregressive transformer modules. To align 2D image features with 1D token sequences, the model uses optimal transport via Sinkhorn algorithm, ensuring efficient 2D-1D mapping. During training, the global tokens are learned first and then frozen, while local aligners are scaled progressively, supporting parallel training. The architecture leverages multi-modal encoders like DINOv2 for semantic and SigLIP for local features, optimizing both reconstruction and generation performance.
Key Results
- On ImageNet-256, with only 64 tokens, the model achieves a gFID of 1.56, outperforming existing 1D tokenizers like TiTok and FlexTok, and approaching 2D methods like VA-VAE. PSNR and SSIM metrics also show superior reconstruction fidelity, validating the compression quality.
- Scaling local aligners from 10M to 600M parameters results in consistent performance gains, reducing training costs by approximately 40% and training time by 54%. The model maintains high generation quality across scales, demonstrating excellent scalability.
- In downstream image generation tasks, the model surpasses baselines such as MAR-H and Lightning-DiT, achieving state-of-the-art results with fewer tokens and less computational overhead, especially without classifier-free guidance.
Significance
This work addresses fundamental inefficiencies in image tokenization by decoupling semantic and detailed information, enabling highly compressed yet expressive representations. It significantly advances the scalability of generative models, reducing computational costs and improving quality simultaneously. The approach opens new avenues for resource-efficient large-scale multimodal systems, impacting fields from image compression to real-time generation and multimodal understanding. Its ability to generate high-fidelity images with minimal tokens demonstrates a breakthrough in balancing compression and quality, crucial for deploying AI in resource-constrained environments.
Technical Contribution
The paper introduces a novel hierarchical tokenization framework combining global-local decomposition with self-supervised prediction and optimal transport alignment. It innovates by decoupling global semantics from local details, enabling scalable multi-modal alignment and parallel training. The multi-codebook design and differentiable soft quantization facilitate end-to-end optimization, while the progressive scaling of local aligners enhances efficiency. These contributions collectively push the boundaries of efficient, high-quality image tokenization, providing a new paradigm for large-scale generative modeling.
Novelty
This is the first work to explicitly decompose image tokens into global and local groups within a self-supervised framework, leveraging multi-modal encoders and optimal transport for 2D-1D alignment. Unlike prior methods that mix all information within tokens, SelfBootTok isolates semantic and detail features, greatly improving scalability and efficiency. Its hierarchical, decoupled design represents a significant departure from traditional monolithic tokenizers, enabling high compression ratios without sacrificing detail, thus setting a new benchmark in image tokenization.
Limitations
- Despite its efficiency, the model's performance degrades slightly on highly complex scenes with extreme compression (fewer than 64 tokens).
- The reliance on pre-trained encoders like DINOv2 and SigLIP introduces dependency on external models, which may limit adaptation to new domains.
- Scaling the local aligner to very large sizes increases training complexity and resource requirements, posing challenges for deployment in resource-constrained environments.
Future Work
Future directions include integrating multi-scale features for better high-resolution performance, exploring unsupervised or weakly supervised local aligners, and extending the framework to video and 3D data. Improving domain adaptation and reducing reliance on pre-trained encoders will further enhance model versatility. Additionally, investigating more efficient optimal transport algorithms could further reduce training costs and improve scalability.
AI Executive Summary
The quest for efficient yet expressive image representations remains a central challenge in computer vision. Traditional tokenization methods often encode all levels of detail within each token, leading to redundancy and limited scalability. This bottleneck hampers the performance of generative models, especially under strict compression constraints. Addressing this, SelfBootTok introduces a hierarchical, global-local decomposition strategy that separates high-level semantic tokens from fine-grained local details. By leveraging self-supervised learning, the model predicts local information solely from global tokens, shifting the complexity away from the generator. This approach significantly reduces computational costs—by approximately 40%—while maintaining or surpassing state-of-the-art quality, achieving a gFID of 1.56 with only 64 tokens on ImageNet-256. The architecture employs multi-modal encoders like DINOv2 for semantic alignment and SigLIP for local features, combined with optimal transport techniques to map 2D image features into a 1D token sequence efficiently. The training process is designed for scalability: global tokens are learned once and frozen, while local aligners are scaled progressively, enabling parallel training and further cost savings. Extensive experiments demonstrate that SelfBootTok not only outperforms existing 1D tokenizers but also approaches the performance of more complex 2D methods, setting new benchmarks in image compression and generation. Its flexible, scalable design paves the way for deploying large-scale multimodal models in resource-constrained settings, marking a significant step forward in efficient visual AI systems.
Deep Dive
Abstract
Despite progress in image tokenization, standard methods encode redundant information by mixing all granularities within each token, thus redundancy persists between tokens. The mix of information of different granularity also complicates the training of generators. This paper introduces SelfBootTok, a method that resolves this by cleanly decomposing information into global and local token groups. Through self-bootstrapped learning, the model predicts local details exclusively from global tokens, shifting the burden of visual details from the generator to the tokenizer. Consequently, our generator is far more efficient, requiring only global tokens and reducing computation by approximately 40%, while delivering superior reconstruction and generation. Moreover, this paradigm scales elegantly: by leveraging more data or parameters to self-supervise local representation learning, SelfBootTok achieves a new state-of-the-art gFID score of 1.56 using only 64 tokens.