Initialization is Half the Battle: Generating Diverse Images from a Guidance Potential Posterior
Li et al. propose DivIn, using Langevin dynamics to optimize initial noise, significantly boosting diversity in diffusion models, outperforming existing methods.
Key Findings
Methodology
DivIn formulates the initial noise as sampling from a guidance potential posterior, re-weighting the standard Gaussian prior toward low-potential regions. It employs Langevin dynamics to actively explore these regions, guided by an energy function U(x, c) derived from the difference between conditional and unconditional denoising estimates. The process involves iterative gradient updates and noise injection, maintaining the data manifold validity. This approach is compatible with diffusion and flow matching models, leading to improved mode coverage and diversity without retraining. Extensive experiments on ImageNet-1K and multi-prompts datasets demonstrate its effectiveness in both class-to-image and text-to-image tasks, with significant gains in diversity metrics like Vendi score and coverage, while preserving high image fidelity and semantic alignment.
Key Results
- On ImageNet-1K, DivIn increased Vendi score to 4.688, surpassing SAIL (4.549) and trajectory-guided methods, with FID reduced to 16.158, indicating better diversity-quality balance.
- In text-to-image generation, DivIn improved CLIP scores to above 1.97, enhanced aesthetic scores, and maintained high semantic fidelity, validating its broad applicability.
- Combining DivIn with trajectory-based methods further expanded the diversity-accuracy Pareto frontier, demonstrating its orthogonality and complementary nature, especially in complex scenarios.
Significance
This work addresses a fundamental limitation in generative models by focusing on the initialization phase, which was previously overlooked. By optimizing the starting point, the method effectively mitigates mode collapse, a long-standing challenge. Its plug-and-play nature and theoretical guarantees make it highly practical for real-world applications like art, virtual environments, and content creation, where diversity is critical. The approach bridges the gap between theoretical understanding of the landscape and practical diversity enhancement, offering a new paradigm for generative modeling.
Technical Contribution
The core technical innovation is the formulation of a guidance potential posterior that biases initial noise toward low-potential regions, combined with Langevin dynamics for efficient sampling. The potential U(x, c) is estimated via the difference in denoising estimates, providing a stable, data-driven measure of guidance intensity. The algorithm maintains the data manifold, avoids mode collapse, and is compatible with various generative architectures. It extends the theoretical understanding of landscape geometry in diffusion models, offering guarantees on diversity improvement without sacrificing quality.
Novelty
This is the first work to explicitly incorporate guidance potential posterior sampling into the initialization phase of diffusion models. Unlike prior methods that focus solely on trajectory guidance or seed optimization, DivIn actively reshapes the initial distribution, leveraging stochastic Langevin updates to explore flat, diverse basins. This approach introduces a new perspective on the role of initialization in mode coverage, setting a foundation for future research on landscape-aware generative strategies.
Limitations
- The accuracy of the guidance potential U(x, c) heavily influences results; estimation errors can reduce diversity gains.
- Langevin sampling may be slow in high-dimensional or highly complex landscapes, increasing computational costs.
- Hyperparameter tuning (e.g., τ, step size) remains sensitive; large-scale deployment requires further optimization for efficiency.
Future Work
Future directions include adaptive tuning of the temperature τ and step size for faster convergence, integrating learned potential functions for better landscape modeling, and extending the approach to multimodal and high-dimensional generation tasks. Exploring reinforcement learning or self-supervised signals to refine the initialization distribution could further enhance diversity. Additionally, applying this strategy to other generative frameworks like GANs or autoregressive models may broaden its impact.
AI Executive Summary
Generative models such as diffusion and flow matching have revolutionized high-fidelity image synthesis, yet they often suffer from mode collapse, limiting their creative diversity. Traditional solutions mainly focus on manipulating the sampling trajectory, adjusting guidance scales or repelling trajectories during the generation process. However, these methods overlook the critical influence of the initial noise distribution, which fundamentally shapes the diversity of generated outputs.
Xiang Li and colleagues introduce DivIn, a novel initialization strategy that actively explores the latent space using Langevin dynamics guided by a potential function U(x, c). This function measures the divergence between conditional and unconditional denoising estimates, effectively biasing the starting point toward low-potential, flat regions of the landscape. By doing so, DivIn encourages the sampling trajectories to explore multiple modes, significantly reducing mode collapse.
The core innovation lies in formulating the initial noise as a sample from a guidance potential posterior, which re-weights the standard Gaussian prior. This approach is computationally efficient, requiring only a few Langevin steps, and is compatible with various diffusion and flow models. Extensive experiments on datasets like ImageNet-1K and diverse prompt collections demonstrate that DivIn outperforms existing initialization and trajectory-based methods in metrics such as Vendi score, coverage, and FID, while maintaining high semantic fidelity.
Moreover, DivIn's orthogonality to trajectory guidance allows it to be combined with other sampling techniques, further expanding the diversity-quality Pareto frontier. Its simplicity, theoretical grounding, and empirical effectiveness make it a promising avenue for advancing generative modeling, especially in applications demanding rich content variation. Future work will explore adaptive potential estimation, faster Langevin sampling, and broader applicability across generative paradigms, aiming to unlock even higher levels of diversity and creativity.
Deep Dive
Glossary
Guidance Potential (引导潜能)
一种衡量生成条件引导强度的能量函数，用于调节初始化噪声的分布，促进多样性。/ A function measuring the strength of guidance in conditional generation, used to bias initial noise towards diverse modes.
在论文中，指导潜能U(x, c)用于优化初始化噪声分布，避免模式崩溃。
Langevin动力学
一种随机梯度采样方法，通过引入噪声和梯度更新，探索目标分布的高潜能区域。/ A stochastic sampling method that combines gradient information and noise to explore probability landscapes.
用于从偏置的潜在后验中采样初始化噪声，增强多样性。
Vendi Score (Vendi分数)
衡量生成样本多样性的指标，反映有效模式数。/ A metric quantifying the diversity of generated samples by estimating the number of effective modes.
在实验中用以评估生成多样性。
潜在能量U(x, c)
基于去噪估计差异的能量函数，反映样本在条件c下的引导强度。/ An energy function derived from denoising estimate differences, indicating guidance strength under condition c.
用于调节初始化偏向低潜能区域。
Flow Matching (流匹配)
一种连续时间生成模型，通过ODE连接数据与噪声分布。/ A generative framework using ordinary differential equations to map data to noise distribution.
与扩散模型互补，用于生成任务。
Open Questions Unanswered questions from this research
- 1 如何设计更精确的潜在能量U(x, c)以适应不同数据分布和任务仍未充分解决，估计偏差可能影响多样性提升效果。
- 2 Langevin采样在高维空间中的收敛速度和效率仍是瓶颈，需开发更快的采样算法或优化策略。
- 3 如何在保持多样性的同时，进一步提升生成质量和语义一致性，是未来研究的重要方向。
Applications
Immediate Applications
艺术内容创作
利用DivIn增强艺术作品的多样性，满足不同风格和主题的需求，适合艺术家和设计师快速生成丰富素材。
虚拟场景生成
在虚拟现实和游戏开发中，使用DivIn生成多样化场景和角色，提升沉浸感和内容丰富度。
Long-term Vision
内容自动化与个性化
结合DivIn实现个性化内容定制，满足不同用户偏好，推动智能内容生成的普及。
跨模态多样性生成
扩展到文本、音频、视频等多模态领域，推动多样化、多场景的自动内容创作与交互。
Abstract
Despite the remarkable fidelity of generative models, they frequently suffer from mode collapse. Existing strategies for enhancing diversity predominantly focus on intervening during the generation trajectory. We identify a critical oversight that the standard Gaussian initialization often causes trajectories to collapse into dominant modes because it is agnostic to the guidance potential landscape. In this work, we formulate selecting the initial noise from a guidance potential posterior, which effectively re-weights the prior towards diversity-rich regions. To sample from this distribution efficiently, we introduce Diversity-inducing Initialization (DivIn), which leverages Langevin dynamics to actively navigate the initialization landscape, steering initial noise away from collapsing regions while anchoring them to the valid data manifold. Our method serves as an inference-time diversity enhancement compatible with both diffusion and flow matching models. Extensive experiments show that DivIn exhibits a superior performance in both class-to-image and text-to-image scenarios. Furthermore, we highlight that as DivIn is orthogonal to trajectory-based methods, combining them significantly expands the diversity-quality Pareto frontier beyond what either achieves in isolation.