Improved Sampling Schedules for Discrete Diffusion Models

Over recent years, discrete diffusion models have gained prominence for their ability to generate sequence data such as text, music, and symbolic representations. Building on early works like Austin et al. (2021) and Lou et al. (2024), these models leverage stochastic processes to gradually transform noise into meaningful data. Unlike autoregressive models, diffusion approaches enable parallel generation and bidirectional refinement, offering advantages in flexibility and controllability. Despite these advances, the core challenge remains: how to optimize the sampling schedule to improve efficiency without sacrificing quality. Existing methods like fixed linear schedules or error-based approaches lack a solid theoretical foundation, often leading to suboptimal resource allocation. The gap between empirical heuristics and physical principles has limited the potential for systematic improvements, especially in complex or high-dimensional settings.

The main issue addressed is the inefficient and heuristic nature of current sampling schedules in discrete diffusion models. These schedules often fail to account for the nonlinear dynamics of information flow during generation, resulting in unnecessary computation during trivial noise regions and insufficient resolution during critical semantic transitions. This imbalance hampers the models' ability to generate high-quality samples efficiently. Moreover, the lack of a unified theoretical framework prevents principled design and analysis of scheduling strategies, constraining the scalability and interpretability of these models. Addressing this problem requires a fundamental understanding of the underlying physics of the reverse process, particularly how information propagates and dissipates in a probabilistic and geometric context.

The paper's core innovation lies in applying thermodynamic concepts—specifically entropy production—to analyze and optimize the reverse diffusion process. It introduces the entropy production rate as a measure of information flow, establishing a link between thermodynamics and optimal transport geometry. This connection enables the derivation of bounds on the Wasserstein distance, which serve as speed limits for the generative process. Based on these insights, two scheduling algorithms are developed: the Entropic Discrete Schedule (EDS), which modulates time steps according to the instantaneous information load, and the Wasserstein Discrete Schedule (WDS), which adjusts steps to maintain constant geometric progress. These schedules are computed using pretrained neural networks that estimate the non-adiabatic entropy, allowing for adaptive, physics-inspired step sizes without additional training. The approach offers a principled, theoretically grounded alternative to heuristic schedules, with demonstrated empirical gains.

• �� Model the reverse diffusion as a thermodynamic system, defining entropy production rate Htot. • Decompose Htot into adiabatic and non-adiabatic components, focusing on the latter as the information cost. • Derive bounds on Wasserstein distance using the total entropy production and system mobility. • Estimate non-adiabatic entropy via pretrained neural networks, using the diffusion trajectory data. • Construct two schedules: EDS based on the instantaneous entropy rate, WDS based on Wasserstein distance. • Map physical progress to intrinsic time via inverse functions, ensuring uniform steps in information or geometry. • Implement the schedules in inference, without retraining, to optimize sampling efficiency.

Experiments span synthetic count data, music notation, CIFAR-10 images, and text generation. All models are pretrained, with the proposed schedules applied during inference. Metrics include error rates, Hellinger distance, sample diversity, and computational cost (NFEs). Comparisons against linear and JYS schedules show consistent improvements, especially in low-NFE regimes. Results demonstrate that WDS at 8 NFEs matches the quality of traditional methods at 32 NFEs, with significant speedups. In musical tasks, EDS achieves near-fidelity at a fraction of the cost. Image and text tasks confirm the robustness and versatility of the schedules across modalities.

WDS at 8 NFEs achieves comparable quality to standard methods at 32 NFEs, a 4× speedup. EDS at 16 NFEs produces samples similar to 512 NFEs in quality. Across tasks, the schedules reduce errors, improve diversity, and lower computational costs. They outperform heuristic and error-based schedules, validating the physical principles guiding their design. These results highlight the potential for physics-inspired scheduling to revolutionize discrete diffusion sampling.

Applicable to sequence, symbol, and multimodal data generation, especially in resource-constrained or real-time scenarios. The physics-based schedules enable faster inference without retraining, suitable for deployment in edge devices, online systems, and large-scale AI applications. The framework can be extended to continuous models and integrated with multi-task learning, broadening its impact across AI domains.

Estimating entropy production relies on pretrained neural networks, which may introduce bias or inaccuracies in complex data. The schedules are discretized, limiting responsiveness to dynamic changes. High-dimensional data pose computational challenges for accurate Wasserstein and entropy estimation. Future work should focus on adaptive, continuous, and more scalable methods to address these issues.