Simplified and Generalized Masked Diffusion for Discrete Data

The evolution of generative modeling has seen a shift from autoregressive methods like GPT to diffusion-based approaches, which excel in continuous domains. Early diffusion models such as DDPM and DDIM demonstrated remarkable image synthesis capabilities, but their extension to discrete data posed challenges due to the non-Euclidean nature of discrete spaces. Recent efforts, including Austin et al.'s masked diffusion, attempted to adapt these models for text and categorical data, yet faced issues with complex objectives, unstable training, and limited scalability. Embedding discrete data into continuous spaces or designing specialized transition matrices partially addressed these issues but introduced new complexities. This paper builds upon these foundations, aiming to unify and simplify the training process while maintaining high performance across modalities.

Despite progress, discrete diffusion models suffer from overly complex formulations, unstable training, and inefficient sampling. Existing methods often rely on heuristic schedules or score-based parameterizations that lack theoretical clarity, leading to suboptimal results and limited scalability. The core challenge is to develop a unified, simple objective that ensures stable training, theoretical consistency, and high-quality generation across diverse data types. Addressing this gap is crucial for advancing discrete data modeling, especially for large-scale NLP and image synthesis tasks where efficiency and robustness are paramount.

The paper's main innovations include: 1) deriving a simplified ELBO as a weighted integral of cross-entropy losses, reducing training complexity; 2) establishing a continuous-time limit that ensures theoretical rigor and consistency; 3) proposing state-dependent masking schedules that adaptively prioritize tokens based on importance, improving generative quality; 4) unifying analysis of various continuous-time discrete diffusion models, clarifying their relationships and parameter effects. These innovations collectively enable more stable, efficient training and higher-quality generation, addressing longstanding issues in discrete diffusion modeling.

• �� Define the forward process as a Markov chain with state transition matrices, extending to continuous time via limit analysis, ensuring theoretical consistency.
• �� Parameterize the reverse process with neural networks predicting the original data distribution conditioned on noisy inputs, using softmax outputs.
• �� Derive the ELBO as a weighted integral of cross-entropy losses over time, simplifying the training objective.
• �� Introduce state-dependent masking schedules, where the probability of masking each token depends on its importance, modeled via functions αt,i.
• �� Optimize the schedule parameters using unbiased gradient estimators like REINFORCE, enabling adaptive importance weighting.
• �� Extend the framework to multi-token data by factorizing the transition matrices and prediction models.
• �� Use ancestral sampling for generation, with schedule-aware unmasking to improve sample quality.

Experiments involved training on datasets including OpenWebText, CIFAR-10, and ImageNet 64×64. The models were evaluated using perplexity, bits per pixel, and FID scores. Hyperparameters included T=256 steps, with linear and cosine schedules. Ablation studies examined the impact of schedule choices and model parameterizations. The models were compared against GPT-2, SEDD, and other baselines, demonstrating superior zero-shot transfer, lower perplexity, and better image quality. The training process was monitored for stability and convergence speed, validating the theoretical advantages of the simplified ELBO and schedule design.

The models achieved perplexities of 102.26 (small) and 68.10 (medium) on OpenWebText, outperforming GPT-2 and prior discrete diffusion models. In image modeling, bits per pixel reached 2.75 on CIFAR-10 and 3.40 on ImageNet 64×64, surpassing autoregressive models of similar size. The introduction of state-dependent schedules notably improved sample quality, with FID scores dropping from 70 to 17 under cosine scheduling. Training was faster and more stable, with models demonstrating strong zero-shot capabilities across tasks. These results confirm the effectiveness of the simplified objective and adaptive scheduling in high-dimensional, discrete data generation.

The proposed models are suitable for high-fidelity text generation, image synthesis, and content completion tasks. They can be integrated into large-scale language models, image editing tools, and multimodal systems requiring efficient discrete data modeling. The flexible scheduling mechanism allows customization for specific data importance or task requirements, making it adaptable for industry applications like chatbots, content creation, and data augmentation. The models' robustness and scalability suggest potential for deployment in real-time systems and resource-constrained environments.

Despite strong results, training large models remains computationally demanding, especially with complex schedules. The schedule parameters depend on task-specific importance estimations, which may limit generalization. The framework's validation is primarily on static datasets; real-world dynamic data scenarios need further exploration. Additionally, the model's performance under adversarial noise or distribution shifts is not yet fully understood, indicating areas for robustness improvement.