Interpolating Discrete Diffusion Models with Controllable Resampling
IDDM introduces controllable resampling via interpolation, improving discrete diffusion sample quality and flexibility.
Key Findings
Methodology
IDDM employs a three-action reverse process—stay, flip, resample—parameterized by λ, controlling stochasticity. It enforces marginal distribution interpolation between prior and data, ensuring edge consistency. The model directly optimizes a variational lower bound without explicit forward noising, enabling flexible inference. During sampling, λ adjusts the balance between information preservation and randomness, yielding high-quality, diverse outputs. Experiments on molecular graphs and language tasks demonstrate superior performance over state-of-the-art discrete diffusion models.
Key Results
- On QM9, IDDM achieves 99.3% validity with FCD surpassing baseline models, confirming faithful data distribution learning.
- In large molecule benchmarks GuacaMol and MOSES, IDDM outperforms DeFoG and DisCo on FCD and validity metrics, showing robustness in complex graph generation.
- In text modeling on LM1B, IDDM attains perplexity of 42.2, outperforming prior discrete diffusion models, with fewer steps maintaining low perplexity, indicating efficiency.
Significance
This work advances discrete generative modeling by overcoming error accumulation and rigidity issues in existing methods. The controllable resampling mechanism enables high-fidelity, diverse, and flexible sample generation, impacting applications from drug discovery to language modeling. It bridges the gap between fixed Markov chains and adaptive sampling, opening avenues for more robust, scalable, and controllable generative systems across multiple modalities. The approach addresses long-standing challenges, setting a new standard for discrete data synthesis.
Technical Contribution
The key innovation is the interpolation-based reverse process with a tunable λ, decoupling training and inference. The model directly learns a marginal distribution that interpolates between prior and data, avoiding explicit forward noising. The theoretical guarantees ensure marginal consistency and controlled stochasticity, enabling flexible trade-offs. This framework extends the scope of discrete diffusion models, offering a new paradigm for high-quality, controllable generation with theoretical rigor and practical efficiency.
Novelty
This is the first work to formulate a reverse process for discrete data via marginal distribution interpolation with a controllable parameter λ. Unlike prior models relying on fixed Markov transitions, IDDM allows dynamic adjustment of stochasticity during inference, enabling better correction and diversity. Its direct optimization of the marginal distribution and flexible sampling strategy represent a significant departure from traditional diffusion frameworks, providing a new theoretical and practical foundation for discrete generative modeling.
Limitations
- Extreme λ values can lead to either insufficient diversity or error accumulation, requiring careful tuning per task.
- Model performance depends heavily on the prediction accuracy of the neural network, which may degrade in complex or out-of-distribution scenarios.
- Current validation is limited to molecular and language data; applicability to other discrete domains remains to be explored.
Future Work
Future directions include extending the framework to multimodal data, integrating reinforcement learning for quality enhancement, and developing adaptive λ schedules. Further research will focus on reducing hyperparameter sensitivity, improving training stability, and applyingIDDM to broader applications such as discrete time series and structured data synthesis.
AI Executive Summary
Generating high-quality discrete data remains a core challenge in artificial intelligence. Existing models like masked diffusion and uniform diffusion each have notable limitations: the former cannot correct early errors, leading to error accumulation, while the latter relies heavily on intermediate latent states, which can produce low-quality samples. To address these issues, this paper introduces IDDM, a novel interpolating discrete diffusion model that leverages a controllable resampling mechanism.
IDDM's key innovation lies in its reverse process, which employs three actions—staying in the current state, flipping to the target, or resampling from a prior—regulated by a parameter λ. This approach enables dynamic control over stochasticity, balancing preservation of learned information with the ability to correct errors. By constraining the marginal distribution to interpolate between prior and data, IDDM ensures edge consistency and mitigates error propagation.
The model is trained by directly optimizing a variational lower bound on the likelihood, without requiring an explicit forward noising process. During inference, the λ parameter can be tuned to produce samples with varying degrees of diversity and fidelity. Extensive experiments on molecular graph datasets (QM9, GuacaMol, MOSES) demonstrate that IDDM outperforms existing discrete diffusion models in validity, FCD, and diversity metrics. Similarly, in text generation tasks, it achieves lower perplexity and higher diversity, with fewer steps, indicating efficiency.
This work significantly advances the field by providing a flexible, theoretically grounded framework for discrete data generation. Its ability to control the trade-off between information retention and randomness opens new possibilities for applications in drug discovery, natural language processing, and beyond. Despite some limitations at extreme parameter settings, the approach offers a promising direction for future research, including multimodal data synthesis and adaptive sampling strategies, promising to reshape discrete generative modeling.
Deep Dive
Glossary
Discrete Diffusion (离散扩散)
一种逐步引入噪声的生成模型，通过逆向过程还原数据。
论文中描述的模型框架基础。
边缘插值 (Marginal Interpolation)
在模型中，将边缘分布定义为先验与数据分布的线性组合。
确保逆向转移边缘一致性的重要机制。
重采样系数 (Resampling Coefficient λ)
调节逆向过程中随机性和信息保持的参数。
模型核心调节参数。
逆向过程 (Reverse Process)
从噪声逐步还原到原始数据的生成过程。
模型训练和采样的关键步骤。
变分下界 (ELBO)
用于训练模型的目标函数，最大化以逼近真实数据分布。
训练过程中优化的目标。
Open Questions Unanswered questions from this research
- 1 如何在极端λ值下保持样本多样性和质量的平衡仍需深入研究。
- 2 模型在复杂或多模态数据中的泛化能力尚未充分验证。
- 3 未来需探索更高效的参数调节和训练策略，以适应不同应用场景。
Applications
Immediate Applications
药物设计
利用IDDM生成具有潜在药效的分子结构，加快新药研发流程。
自然语言处理
提升文本生成的多样性和准确性，用于聊天机器人、内容创作等。
Long-term Vision
多模态内容生成
结合图像、视频、文本等多模态数据，推动智能内容创作。
Abstract
Discrete diffusion models form a powerful class of generative models across diverse domains, including text and graphs. However, existing approaches face fundamental limitations. Masked diffusion models suffer from irreversible errors due to early unmasking, while uniform diffusion models, despite enabling self-correction, often yield low-quality samples due to their strong reliance on intermediate latent states. We introduce IDDM, an Interpolating Discrete Diffusion Model, that improves diffusion by reducing dependence on intermediate latent states. Central to IDDM is a controllable resampling mechanism that partially resets probability mass to the marginal distribution, mitigating error accumulation and enabling more effective token corrections. IDDM specifies a generative process whose transitions interpolate between staying at the current state, resampling from a prior, and flipping toward the target state, while enforcing marginal consistency and fully decoupling training from inference. We benchmark our model against state-of-the-art discrete diffusion models across molecular graph generation as well as text generation tasks, demonstrating competitive performance.