Generate in Reconstruction Space, Match in Semantic Space: Transport Geometry for One-Step Generation
Proposes semantic space matching using SSL features with Sinkhorn divergence, reducing ImageNet FID by 39×, improving one-step generative quality.
Key Findings
Methodology
This work employs frozen SSL features to define a semantic space where distribution matching is performed via Sinkhorn divergence. The generator operates in a reconstruction space (e.g., VAE latent), while the matching occurs in a structured semantic feature space, reducing complexity. The approach compares different SSL families (DINOv3, Inception) and introduces a stability metric for transport estimation, guiding feature selection. The framework aligns the training and evaluation metrics, mitigating metric hacking issues and enhancing stability. The core innovation is the separation of generation and matching spaces, leveraging semantic structure for more efficient distribution alignment.
Key Results
- On ImageNet, using SSL features in semantic space reduces FID from 134.56 to 3.46 (MAE 50%), a 39× improvement. DINOv3 features outperform Inception in stability and quality, validating the semantic structure advantage.
- Matching stability correlates strongly with generation quality; selecting features based on stability yields better samples. The method bridges the gap between one-step generation and high-quality synthesis.
- Introducing a quantitative stability metric (˜DN) allows effective feature filtering, leading to consistent improvements across SSL families and training setups.
Significance
This research demonstrates that leveraging structured semantic features significantly enhances one-step generative modeling, addressing longstanding issues of high-dimensional distribution matching. It offers a theoretical and practical framework for improving sample quality, stability, and interpretability. The findings challenge the conventional reliance on metrics like FID alone, emphasizing the importance of matching stability. The approach paves the way for more robust, efficient, and semantically aware generative systems, with broad implications for content creation, virtual reality, and AI-driven design.
Technical Contribution
The paper introduces a dual-space framework combining a reconstruction space for generation and a semantic feature space for distribution matching. It employs Sinkhorn divergence as a tractable approximation of Wasserstein distance, coupled with a novel stability metric for transport estimation. The methodology integrates SSL pretraining, feature space geometric analysis, and classifier-free guidance into a unified pipeline, enabling stable, high-fidelity one-step generation. Theoretical insights into the statistical estimation of OT maps underpin the feature selection process, advancing the understanding of transport-based generative training.
Novelty
This work is the first to systematically incorporate structured SSL features into a dual-space distribution matching framework for one-step generation. It emphasizes the geometric and statistical properties of feature spaces, introducing a stability-based criterion for feature selection. Unlike prior methods that use SSL features as auxiliary regularizers, this approach directly optimizes in a semantically meaningful space, significantly improving efficiency and quality. The explicit focus on transport estimation stability and metric alignment distinguishes it from existing literature.
Limitations
- The approach depends heavily on the quality of pretrained SSL models; poor semantic features limit performance. In some domains, suitable SSL models may be unavailable.
- Transport estimation stability, while improved, still faces challenges in highly complex or high-dimensional scenarios, potentially affecting robustness.
- Computational costs remain significant due to feature extraction and Sinkhorn iterations, especially at large scales. Further optimization is needed for real-time applications.
Future Work
Future research will explore multi-scale and multi-layer SSL features to capture richer semantics. Extending the framework to video and 3D data, integrating multi-modal cues, and improving computational efficiency are key directions. Additionally, developing adaptive feature selection strategies based on task-specific criteria and further theoretical analysis of transport stability will enhance robustness and applicability.
AI Executive Summary
Generative modeling has seen rapid advancements, yet high-dimensional distribution matching remains a core challenge. Traditional pixel-based approaches struggle with complex geometries and semantic nuances, limiting the quality and stability of generated images. Recent efforts leverage pretraining and feature-based metrics, but often overlook the geometric structure of feature spaces. This paper introduces a novel framework that separates generation and matching into two distinct spaces: a reconstruction space for sample synthesis and a structured semantic feature space for distribution alignment. The key innovation lies in utilizing frozen SSL features—such as DINOv3 and Inception—to define a geometry that better captures semantic content, thus simplifying the matching problem.
The authors employ Sinkhorn divergence, a computationally tractable approximation of Wasserstein distance, to perform distribution matching in the semantic space. This approach significantly reduces the difficulty of transport estimation, leading to more stable training and higher-quality samples. Extensive experiments on ImageNet demonstrate that this method reduces FID from 134.56 to 3.46, a 39-fold improvement, surpassing many existing single-step models. The study also reveals that the stability of transport estimation correlates strongly with generation quality, and that selecting features based on this criterion yields optimal results.
Beyond empirical gains, the work offers a theoretical contribution by analyzing the statistical properties of transport map estimation, providing a quantitative metric (˜DN) for feature quality. It highlights that the best features for training do not necessarily match those used for evaluation, exposing potential metric hacking issues. Overall, this research advances the understanding of how semantic structure and transport geometry influence one-step generation, opening new avenues for efficient, high-fidelity image synthesis with broad industrial and academic implications.
Deep Analysis
Background
The evolution of generative models, from GANs to diffusion techniques, has significantly improved image synthesis. Self-supervised learning (SSL) pretraining, exemplified by models like DINO, has enhanced semantic understanding, benefiting downstream tasks. Prior work has integrated SSL features as regularizers or auxiliary signals, but their role in distribution matching remains underexplored. Optimal transport (OT), especially Sinkhorn divergence, offers a promising theoretical foundation for distribution alignment, yet practical challenges like instability and high computational costs limit its application. This paper addresses these issues by leveraging the semantic structure of SSL features to improve OT-based matching, aiming for more stable, efficient, and semantically meaningful generation.
Core Problem
High-dimensional distribution matching in generative modeling faces geometric complexity, noise interference, and computational inefficiency. Traditional pixel-space methods lack semantic robustness, leading to poor sample quality and training instability. Existing feature-based approaches often ignore the geometric structure of the feature space, resulting in suboptimal matching and metric hacking. The core challenge is to develop a method that exploits semantic structure to stabilize OT estimation, reduce complexity, and improve sample fidelity, especially in one-step generation scenarios where efficiency and stability are critical.
Innovation
The paper introduces a dual-space framework: generation occurs in a reconstruction space (e.g., VAE latent), while distribution matching is performed in a structured semantic feature space. It employs Sinkhorn divergence as a differentiable, computationally feasible transport metric. A key innovation is the stability metric (˜DN), quantifying the reliability of OT estimates, guiding feature selection. The approach combines SSL pretraining, geometric analysis, and classifier-free guidance, enabling high-quality, stable one-step generation. This integration of transport geometry and semantic structure is a significant departure from prior work, which often treats features as auxiliary or ignores geometric considerations.
Methodology
- �� Generate samples in a low-dimensional reconstruction space (e.g., VAE latent) from noise prior.
- �� Use a frozen SSL feature extractor (e.g., DINOv3, Inception) to encode both generated and real samples, avoiding pixel-level comparisons.
- �� Define the transport cost as squared Euclidean distance in the feature space.
- �� Compute Sinkhorn divergence between generated and real feature distributions, approximating Wasserstein distance.
- �� Optimize generator parameters to minimize this divergence, ensuring distributional alignment.
- �� Introduce a stability metric (˜DN) based on the variance of Sinkhorn couplings across minibatches, quantifying OT estimation reliability.
- �� Incorporate classifier-free guidance by interpolating between conditional and unconditional distributions during training.
- �� Evaluate different SSL families, analyze the correlation between stability and quality, and select features accordingly.
Experiments
Experiments on ImageNet at 256×256 resolution employ a VAE-based generator (DriftDiT-B/2) trained over 400k steps with AdamW optimizer. The framework compares SSL features like DINOv3, Inception, and MAE, measuring FID, Frechet distance, and transport stability. The results demonstrate that semantic features drastically reduce FID, with DINOv3 outperforming Inception in stability and quality. Ablation studies analyze the impact of feature choice, stability metrics, and guidance weights. The experiments validate that better transport stability correlates with higher sample fidelity, confirming the theoretical insights. The framework also assesses the effect of feature distillation and geometric alignment.
Results
Using SSL features in semantic space reduces ImageNet FID from 134.56 to 3.46, a 39× improvement. DINOv3 features outperform Inception in stability and sample quality, with lower FD and higher matching reliability. The stability metric (˜DN) effectively predicts generation performance, guiding feature selection. The experiments confirm that stable transport estimation is crucial for high-quality one-step generation, and that optimizing for geometric alignment alone can lead to metric hacking, emphasizing the importance of stability-based criteria.
Applications
This method enables efficient, high-fidelity image synthesis suitable for content creation, virtual environments, and AI-assisted design. It requires pretrained SSL models and a low-dimensional latent space, making it adaptable to various domains. Its robustness and stability make it promising for real-time applications, personalized content, and scalable deployment in industry. Future integration with multi-modal data could further expand its utility.
Limitations & Outlook
Dependence on pretrained SSL models limits applicability in domains lacking suitable models. Transport estimation stability, while improved, may still falter in highly complex scenes. Computational costs for feature extraction and Sinkhorn iterations remain significant, necessitating further optimization. The approach's effectiveness hinges on the semantic quality of SSL features, which may vary across tasks and datasets.
Plain Language Accessible to non-experts
想象你在一家工厂里，工厂每天生产各种商品。以前，工厂用普通的生产线，商品质量不稳定，难以满足客户需求。后来，工厂引入了一套智能检测系统，这个系统能识别商品的核心特征，比如颜色、形状和用途，而不是每个细节都检查。这样，工厂就能更快、更准确地调整生产流程，确保商品符合客户的期望。类似的，论文中用一种叫“语义空间”的方法，把生成的图片和真实图片的“主要内容”匹配起来，而不是在像素层面逐一比对。这样，生成的图片更符合“核心需求”，质量大大提高。这个方法就像用智能检测系统帮工厂提升商品质量一样，帮助生成模型更好地理解和复制真实世界的内容。
ELI14 Explained like you're 14
想象你在玩拼图游戏，你需要把碎片拼成一幅完整的画。以前的方法是看每个碎片的颜色和形状，然后一块一块拼，特别难，因为碎片很多，细节也很多。现在，有个聪明的朋友告诉你，只要关注拼图的“主要内容”，比如人物、背景和主要颜色，就能更快拼好。论文里的方法也是这样：它用一种叫“语义空间”的工具，把图片的“主要内容”提取出来，然后在这个空间里匹配生成的图片和真实图片。这样，模型不用在每个细节上较劲，而是专注于“核心内容”，让生成的图片既快又好看。就像用这个聪明的朋友帮你拼图一样，模型变得更聪明，也更擅长创造高质量的图片。
Abstract
Generative modeling and self-supervised representation learning (SSL) optimize structurally different objectives: generative training rewards distributional fidelity, while SSL rewards semantic coherence. Yet recent work repeatedly finds that SSL features improve generative training, though the mechanism of this synergy remains unclear. Here, we study the benefits of SSL in generative modeling in the framework of one-step generation where the role of representation is explicit: frozen SSL features are used to match generated samples to real data. We use the Sinkhorn divergence in that feature space, providing a tractable surrogate for the Wasserstein distance, the population-level discrepancy approximated by Fréchet-style evaluation metrics (such as FID). We find that this objective becomes highly effective when computed in a semantically structured SSL feature space (a 39$\times$ reduction in ImageNet FID). We trace this behavior primarily to matching estimation: semantic SSL features that suppress nuisance reconstruction details induce a more compact geometry, making distribution matching more tractable. As a consequence, the best training SSL features need not match the features used by the evaluation metric. In particular, we show that using Inception as the feature extractor can improve FID while degrading matching stability and sample quality, revealing a form of metric hacking. Using extensive experiments on ImageNet, we identify which SSL feature families lead to best generation performance and show that matching stability is a quantitative criterion for selecting them. Code is available at https://github.com/Genentech/semantic-transport-generation.