DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents
DisCo-Diff integrates discrete latent variables into diffusion models, reducing ODE curvature and improving image generation, achieving SOTA FID 1.65 on ImageNet-128.
Key Findings
Methodology
DisCo-Diff augments continuous-time diffusion models by learning m-dimensional discrete latents via an encoder, trained end-to-end. Discrete latents encode high-level data structure, simplifying the noise-to-data mapping, reducing the curvature of the generative ODE. A small codebook of few discrete variables is used, with a transformer autoregressively modeling their distribution. The framework employs Gumbel-Softmax relaxation for differentiability, and integrates discrete latents into the denoiser via cross-attention. Experiments on toy, image, and molecular tasks show consistent performance gains, with state-of-the-art results on ImageNet-64/128.
Key Results
- On class-conditioned ImageNet-64/128, DisCo-Diff achieves FID 1.65 with an ODE sampler, outperforming EDM (6.20) and VDM (1.43).
- In toy 2D Gaussian mixtures, discrete latents capture modes, leading to straightened ODE trajectories and lower curvature.
- In molecular docking, discrete latents identify critical atoms, improving multimodal pose modeling.
- Ablation studies confirm that few discrete latents suffice for performance boosts, validating the simplicity and effectiveness of the approach.
Significance
This work addresses the core challenge of modeling complex, multimodal data distributions with diffusion models. By introducing discrete latents, it reduces the complexity of the generative ODE, enabling higher quality synthesis without reliance on large spatial latent grids or pre-trained encoders. The approach enhances both the stability and diversity of generated samples, broadening the applicability of diffusion models in fields like image synthesis and drug discovery. It paves the way for more efficient, scalable, and interpretable generative frameworks that better reflect the inherent structure of real-world data.
Technical Contribution
DisCo-Diff's key innovation lies in combining a small set of learnable discrete latents with continuous Gaussian latent variables, trained jointly with the diffusion model. It employs Gumbel-Softmax for differentiable sampling of discrete variables and models their distribution autoregressively. This separation of discrete and continuous variations simplifies the score matching task, reduces the curvature of the generative ODE, and improves training stability. The framework's end-to-end design, without reliance on pre-trained encoders, offers broad applicability and superior performance, setting new state-of-the-art benchmarks.
Novelty
This is the first systematic integration of few global discrete latent variables into continuous-time diffusion models to explicitly capture multimodal data structures. Unlike prior works relying on large spatial latent grids or pre-trained encoders, DisCo-Diff employs a minimal set of discrete latents, learned jointly with the diffusion process, to significantly reduce ODE curvature and improve sample quality. Its end-to-end training and simple autoregressive modeling of discrete latents represent a novel paradigm in generative modeling, bridging the gap between discrete and continuous latent spaces.
Limitations
- The model's performance may decline on highly complex, high-dimensional multimodal data requiring larger or hierarchical discrete latent spaces.
- Gumbel-Softmax relaxation introduces bias, potentially limiting the discrete latent expressiveness.
- Computational costs remain significant, especially for large-scale datasets or high-resolution images, necessitating further optimization.
Future Work
Future directions include scaling the number of discrete latents, exploring hierarchical structures, integrating pre-trained encoders for richer features, and applying the framework to video and 3D data. Improving sampling efficiency and extending the approach to broader multimodal tasks will further enhance its practical impact.
AI Executive Summary
Diffusion models (DMs) have revolutionized generative learning, enabling high-fidelity image and molecular synthesis. However, their reliance on a single continuous latent variable to encode complex, multimodal data often results in highly curved generative ODE trajectories, making training and sampling challenging. Traditional approaches, such as large spatial latent grids or pre-trained encoders, attempt to address this but introduce complexity and limit generality.
In response, Xu et al. propose DisCo-Diff, a novel framework that introduces a small set of learnable discrete latent variables into the diffusion process. These discrete latents encode high-level, global information about the data, effectively partitioning the complex data distribution into simpler subspaces. The model employs an encoder to infer these discrete latents, trained jointly with the diffusion model using a Gumbel-Softmax relaxation for differentiability. An autoregressive transformer models the distribution over the discrete latents, enabling effective sampling.
The core idea is that discrete latents capture the macro-structure of data, such as global style or class features, reducing the curvature of the generative ODE and simplifying the denoising task. This leads to more stable training, higher sample quality, and better generalization. Extensive experiments on toy datasets, ImageNet, and molecular docking tasks demonstrate consistent improvements, with state-of-the-art FID scores of 1.65 on ImageNet-128, outperforming prior models.
DisCo-Diff's architecture is flexible, end-to-end trainable, and does not rely on pre-trained encoders, making it broadly applicable. Its ability to effectively model multimodal data opens new avenues for high-quality, diverse generative modeling across multiple domains. Despite some limitations in handling extremely complex data, the framework sets a new standard for diffusion-based generative models, promising significant impact in AI research and industry applications.
Deep Analysis
Background
扩散模型在图像和分子结构生成中取得巨大成功，代表作有Ho et al.（2020）和Karras et al.（2022）的工作。这些模型通过逐步添加噪声，学习噪声到数据的映射，极大提升了生成质量。然而，复杂多模态数据的生成仍面临挑战，主要源于单一连续潜变量难以表达多样性，导致生成轨迹弯曲，训练不稳定。近年来，结合离散潜变量的研究逐渐兴起，如VQ-VAE（van den Oord et al., 2017）和空间潜变量模型，试图捕获全局结构，但多依赖大规模空间潜变量和预训练编码器，限制了模型的普适性和效率。
Core Problem
传统连续潜变量模型在多模态数据生成中存在轨迹弯曲、训练不稳定的问题。尤其在高维复杂场景中，单一潜变量难以捕获多样性，导致模型泛化差、样本多样性不足。现有方法多依赖大规模空间潜变量或预训练编码器，增加了模型复杂度和训练成本。如何设计简洁、有效的潜变量结构，既能捕获全局多模态信息，又能简化学习过程，成为亟待解决的核心问题。
Innovation
DisCo-Diff的创新在于引入少量可学习的离散潜变量，结合端到端训练，显著降低生成ODE的弯曲度。具体包括：• 采用离散潜变量编码全局信息，简化噪声到数据的映射• 端到端训练，无需预训练编码器，增强模型普适性• 利用Gumbel-Softmax实现离散潜变量的连续松弛，支持梯度反传• 自回归Transformer建模离散潜变量的联合分布，确保采样多样性。这些创新使模型在复杂多模态任务中表现优异，突破了传统连续潜变量的局限。
Methodology
- �� 设计扩散模型，加入m维离散潜变量z，编码全局信息• 编码器Eϕ输入清晰数据，推断离散潜变量，输出类别分布• 训练目标扩展为同时优化扩散模型和编码器，最小化噪声还原误差• 使用Gumbel-Softmax实现离散潜变量的连续松弛，支持梯度反传• 训练中随机用null潜变量，增强鲁棒性• 自回归模型Aψ学习离散潜变量的联合分布，采样时先生成离散潜变量，再通过ODE采样生成数据• 在网络中加入交叉注意机制，将离散潜变量融入主网络，增强全局特征表达。
Experiments
在Toy二维高斯混合、ImageNet图像合成和分子对接任务中验证模型性能。使用FID作为主要指标，比较不同潜变量数量和架构。Toy数据中，离散潜变量成功捕获模态，ODE轨迹变直。图像任务中，DisCo-Diff实现FID 1.65，优于EDM（6.20）和VDM（1.43）。消融实验验证少量潜变量的有效性。模型在不同采样策略和架构下表现稳定，验证了设计的合理性。
Results
在ImageNet-128类别条件下，DisCo-Diff利用ODE采样实现了FID 1.65，优于现有最优模型（如EDM 2.36和VDM 1.43）。Toy二维高斯混合验证中，离散潜变量成功捕获模态，ODE轨迹变直。分子对接中，离散潜变量帮助识别关键原子，改善多模态不确定性建模。不同架构和潜变量数量的消融实验确认少量潜变量已足够提升性能，验证了设计的有效性。
Applications
该模型适用于高质量图像生成、药物设计和多模态内容创作。无需预训练编码器，端到端训练简化流程，适合多行业应用。未来可扩展到视频、3D模型等场景，推动内容创作和药物研发的革新。模型的全局潜变量有助于捕获数据宏观结构，实现更丰富多样的生成内容。
Limitations & Outlook
模型在极端复杂、多模态数据中潜变量容量可能不足，需扩展潜变量空间。Gumbel-Softmax引入偏差，影响潜变量表达。训练成本较高，尤其在高分辨率任务中需优化推理效率。未来应探索更大潜变量集和高效采样算法，以应对更复杂的场景。
Plain Language Accessible to non-experts
想象你在厨房做饭，食材代表数据。传统做法用一种调料（连续潜变量）调味，但难以表达所有味道，尤其菜多时。DisCo-Diff像加入几种特殊调料（离散潜变量），每种代表一种大致风格，比如辣或甜。厨师（模型）根据这些调料选择不同做法，做出多样菜肴。离散调料让整体味道更易控制，做饭更快更好吃。模型学会这些调料组合，能还原复杂菜肴的风味，就像生成多样图片或分子结构一样。
ELI14 Explained like you're 14
想象你在玩拼图游戏，每块拼图代表一部分图片。以前用一块大拼图拼出整个图，但太复杂容易出错。DisCo-Diff像用几块小拼图，每块代表一个大概主题，比如颜色或形状。这样拼图变简单！模型先学会每块拼图内容，再拼出完整图片。不同拼图组合能拼出很多不同的场景，就像用不同主题拼出不同画面。这让拼图更快、更准，还能拼出丰富多彩的画面！
Abstract
Diffusion models (DMs) have revolutionized generative learning. They utilize a diffusion process to encode data into a simple Gaussian distribution. However, encoding a complex, potentially multimodal data distribution into a single continuous Gaussian distribution arguably represents an unnecessarily challenging learning problem. We propose Discrete-Continuous Latent Variable Diffusion Models (DisCo-Diff) to simplify this task by introducing complementary discrete latent variables. We augment DMs with learnable discrete latents, inferred with an encoder, and train DM and encoder end-to-end. DisCo-Diff does not rely on pre-trained networks, making the framework universally applicable. The discrete latents significantly simplify learning the DM's complex noise-to-data mapping by reducing the curvature of the DM's generative ODE. An additional autoregressive transformer models the distribution of the discrete latents, a simple step because DisCo-Diff requires only few discrete variables with small codebooks. We validate DisCo-Diff on toy data, several image synthesis tasks as well as molecular docking, and find that introducing discrete latents consistently improves model performance. For example, DisCo-Diff achieves state-of-the-art FID scores on class-conditioned ImageNet-64/128 datasets with ODE sampler.