Learned Reference-based Diffusion Sampling for multi-modal distributions

The challenge of sampling from multi-modal distributions has long hindered progress in Bayesian inference, generative modeling, and physical simulations. Early methods like MCMC and Langevin dynamics faced difficulties in high dimensions due to slow mixing and mode trapping. Recent advances in diffusion models, such as score-based diffusion and DDPM, have achieved remarkable results in image synthesis and probabilistic inference. Nonetheless, these models often require careful hyperparameter tuning and struggle with capturing all modes in complex landscapes. Variational inference approaches, including normalizing flows and extended path measures, have attempted to address these issues but still face limitations in multi-modality and tuning sensitivity. This context motivates the development of methods that incorporate prior knowledge to improve sampling robustness and efficiency.

The core problem is effectively sampling from multi-modal distributions with unknown or complex energy landscapes. Existing methods like MCMC suffer from high autocorrelation and mode-switching delays, especially in high dimensions. Variational methods often collapse modes or require extensive tuning of hyperparameters such as noise scales or temperature parameters. Even with known mode locations, accurately capturing their proportions and exploring the full landscape remains difficult. These issues hinder reliable Bayesian inference and generative modeling in complex scenarios, necessitating novel approaches that leverage prior information to guide sampling without excessive tuning or computational costs.

This work introduces LRDS, which uniquely leverages prior mode location knowledge to learn a reference diffusion model aligned with the target distribution’s multi-modality. Unlike traditional approaches relying on fixed Gaussian references, LRDS employs Gaussian mixture models or energy-based neural networks, adaptively capturing the distribution’s structure. The key innovation is the integration of variational path measure optimization using LV divergence, which enhances mode exploration and prevents mode collapse. Additionally, the method combines EM and maximum likelihood techniques for efficient parameter estimation, ensuring tractability and robustness. Theoretically, the authors establish convergence guarantees, and empirically, they demonstrate superior performance in high-dimensional, multi-modal settings, marking a significant step forward in diffusion-based sampling.

• �� Obtain prior mode locations via MCMC (e.g., MALA) initialized at known modes.
• �� Fit a reference distribution (GMM or energy model) using EM or ML, based on samples from these locations.
• �� Define a variational diffusion process guided by the learned reference scores, optimizing the LV divergence to improve mode coverage.
• �� Implement the process in discrete time with Euler-Maruyama or exponential integrators, updating parameters iteratively.
• �� After training, generate samples by simulating the diffusion process from the learned reference model, ensuring multi-modal coverage.
• �� Use the learned model to guide the reverse diffusion, capturing all significant modes with minimal tuning.

Experiments involve synthetic high-dimensional Gaussian mixtures and real-world complex distributions. Metrics include mode proportion error, sample quality (FID), and mode coverage rate. Baselines include traditional diffusion models, MCMC, and variational methods. Hyperparameters such as number of modes, step size, and noise scales are tuned to evaluate robustness. Ablation studies compare GMM and energy-based references, demonstrating the importance of prior knowledge. Results show LRDS achieves lower mode error (<0.05), higher coverage (>95%), and improved sample fidelity, confirming its effectiveness in multi-modal high-dimensional sampling.

LRDS accurately estimates mode proportions with errors below 0.05, outperforming baseline methods by over 30% in complex distributions. It captures more than 95% of modes consistently, with sample quality metrics surpassing previous methods. The energy-based variant further improves results in challenging scenarios, successfully identifying multiple local maxima. The method exhibits robustness to prior inaccuracies, maintaining high performance even with biased mode estimates. These results validate LRDS’s capability to handle high-dimensional, multi-modal distributions efficiently and reliably.

LRDS can be directly applied to Bayesian posterior sampling, molecular dynamics, and generative modeling tasks requiring multi-modal exploration. Its ability to incorporate prior mode information reduces tuning efforts and enhances sampling stability. The method is suitable for high-dimensional problems where traditional MCMC fails or is computationally prohibitive. Future applications include large-scale scientific simulations, probabilistic programming, and AI systems requiring robust inference in complex landscapes.

The approach relies on reasonably accurate prior mode locations; significant errors can impair sampling quality. Training energy-based models remains computationally intensive, especially in very high dimensions. In extremely complex or poorly understood distributions, some modes may be missed, and the method’s performance depends on the quality of prior information. Further research is needed to automate prior estimation and improve scalability.