GPSToken: Gaussian Parameterized Spatially-adaptive Tokenization for Image Representation and Generation
GPSToken employs Gaussian parameterization for spatially adaptive image tokenization, achieving state-of-the-art FID=1.50 in image generation with 128 tokens.
Key Findings
Methodology
GPSToken utilizes an entropy-driven algorithm to partition images into texture-homogeneous regions, initializing Gaussian parameters (mean for position, covariance for shape). A dedicated transformer refines these parameters, enabling continuous spatial adjustment and content-aware feature extraction. During decoding, differentiable splatting-based rendering reconstructs tokens into 2D feature maps, integrating seamlessly with standard decoders. The model decouples spatial layout from texture features, supporting a two-stage generation: layout synthesis and texture refinement, which simplifies training and enhances quality.
Key Results
- On 128 tokens, GPSToken achieves rec.FID=0.65, PSNR=24.06, SSIM=0.657 in image reconstruction, outperforming existing methods. For image generation, FID reaches 1.50, surpassing recent models like TiTok and FlexTok, demonstrating superior quality.
- Across token counts (64, 128, 256), performance remains robust, with 256 tokens yielding the best results, validating the effectiveness of spatial adaptivity and Gaussian modeling.
- The shape-texture decomposition enables a two-stage process, first generating layout parameters, then textures, reducing complexity and improving output fidelity, confirmed by ablation studies.
Significance
This work advances image representation by introducing Gaussian parameterization, allowing flexible, continuous modeling of complex regions. Decoupling structure and texture aligns with human visual perception, facilitating efficient, high-quality image synthesis. The approach addresses limitations of rigid grid-based methods, offering a scalable, interpretable framework that benefits tasks like image compression, editing, and scene synthesis, with broad industry implications.
Technical Contribution
The paper presents a novel Gaussian parameterization framework, combining entropy-driven region partitioning with transformer-based refinement of shape and position parameters. The differentiable splatting renderer bridges the gap between parametric tokens and feature maps, enabling end-to-end training. The shape-texture decoupling supports a two-stage generation pipeline, simplifying training and improving quality. Compared to prior grid or segmentation-based methods, GPSToken offers a lightweight, expressive, and flexible representation, pushing the frontier of spatially adaptive image modeling.
Novelty
This is the first work to integrate Gaussian parameterization with spatial adaptivity for image tokens, moving beyond fixed grids or segmentation maps. It introduces a continuous, differentiable model that captures complex shapes and textures, enabling precise, flexible, and efficient image encoding and generation. The shape-texture decoupling for two-stage synthesis is a key innovation, setting a new paradigm in visual modeling.
Limitations
- The Gaussian model's linear assumptions may struggle with highly non-linear or extremely irregular regions, limiting representation fidelity in some scenarios.
- Region partitioning based on entropy can be sensitive to noise or boundary ambiguity, potentially affecting the accuracy of initial Gaussian parameters.
- High-resolution images increase computational load, especially during rendering and refinement, requiring further optimization for real-time applications.
Future Work
Future directions include multi-scale Gaussian modeling to better capture hierarchical structures, integrating multi-modal data for richer scene understanding, and optimizing rendering for real-time deployment. Enhancing robustness in complex scenes and extending to video domains are also promising avenues.
AI Executive Summary
In the rapidly evolving field of computer vision, the challenge of representing and generating complex images with high fidelity remains a central focus. Traditional approaches, such as grid-based tokenization, impose rigid structures that limit flexibility in capturing diverse shapes and textures. These methods often lead to redundant representations and struggle with irregular regions, hindering the quality of reconstruction and synthesis.
Addressing these limitations, the paper introduces GPSToken, a novel framework that employs Gaussian parameterization to achieve spatially adaptive, non-uniform image tokenization. By leveraging an entropy-driven algorithm, the image is partitioned into texture-homogeneous regions of varying sizes. Each region is modeled as a 2D Gaussian, with parameters encoding position, shape, and orientation, coupled with texture features. A specialized transformer refines these parameters, enabling continuous spatial adjustment and content-aware feature extraction. During decoding, a differentiable splatting-based renderer reconstructs tokens into feature maps, facilitating seamless integration with standard decoders for end-to-end training.
The core innovation lies in the decoupling of spatial layout and texture features, supporting a two-stage generation process. First, a lightweight network synthesizes the Gaussian parameters representing the overall structure. Then, a diffusion-based model generates detailed textures conditioned on these parameters, ensuring structural consistency and high visual fidelity. Extensive experiments on ImageNet demonstrate that GPSToken achieves state-of-the-art results, with a reconstruction FID of 0.65 and a generation FID of 1.50 using only 128 tokens. These results surpass existing methods like TiTok, FlexTok, and MaskGIT, validating the effectiveness of the shape-texture decomposition and spatial adaptivity.
This work significantly advances the field by providing a flexible, efficient, and interpretable image representation framework. Its ability to model complex regions with minimal redundancy opens new avenues for high-quality image compression, editing, and synthesis. The decoupled two-stage approach simplifies training and improves generation quality, making it highly suitable for practical applications in AI-driven visual content creation. Despite its strengths, challenges remain in handling highly irregular regions and reducing computational costs at high resolutions. Future research will focus on multi-scale modeling, real-time rendering, and broader scene understanding, promising a transformative impact on intelligent visual systems.
Deep Dive
Limitations & Outlook
What gaps remain?
Abstract
Effective and efficient tokenization plays an important role in image representation and generation. Conventional methods, constrained by uniform 2D/1D grid tokenization, are inflexible to represent regions with varying shapes and textures and at different locations, limiting their efficacy of feature representation. In this work, we propose $\textbf{GPSToken}$, a novel $\textbf{G}$aussian $\textbf{P}$arameterized $\textbf{S}$patially-adaptive $\textbf{Token}$ization framework, to achieve non-uniform image tokenization by leveraging parametric 2D Gaussians to dynamically model the shape, position, and textures of different image regions. We first employ an entropy-driven algorithm to partition the image into texture-homogeneous regions of variable sizes. Then, we parameterize each region as a 2D Gaussian (mean for position, covariance for shape) coupled with texture features. A specialized transformer is trained to optimize the Gaussian parameters, enabling continuous adaptation of position/shape and content-aware feature extraction. During decoding, Gaussian parameterized tokens are reconstructed into 2D feature maps through a differentiable splatting-based renderer, bridging our adaptive tokenization with standard decoders for end-to-end training. GPSToken disentangles spatial layout (Gaussian parameters) from texture features to enable efficient two-stage generation: structural layout synthesis using lightweight networks, followed by structure-conditioned texture generation. Experiments demonstrate the state-of-the-art performance of GPSToken, which achieves rFID and FID scores of 0.65 and 1.50 on image reconstruction and generation tasks using 128 tokens, respectively. Codes and models of GPSToken can be found at $\href{https://github.com/xtudbxk/GPSToken}{https://github.com/xtudbxk/GPSToken}$.