Simplex Relaxation for Discrete Diffusion

Discrete diffusion models have made significant progress in the generative field in recent years. Traditional generative models often rely on autoregressive methods, while discrete diffusion models offer a new parallel prediction path. The corruption kernel is the core design choice of discrete diffusion models, determining the intermediate state space and semantics of reverse updates. Existing models implement this design choice in various ways, such as masked diffusion and uniform diffusion.

Challenges remain in the training objectives and reverse sampling processes of discrete diffusion models. Standard uniform diffusion expresses reverse updates directly through sampled categorical states, which preserves the discrete generative process but also means that both training and sampling are formulated through categorical intermediate states. The open question is how to introduce auxiliary probabilistic structures around these transitions to derive tractable objectives and samplers without changing the forward process itself.

Simplax enhances the training objectives and reverse sampling processes of discrete diffusion models by introducing simplex-valued variables. This method retains the original categorical corruption process while introducing an auxiliary simplex-valued variable that couples each corrupted categorical state with a simplex variable. This augmentation yields a tractable Rao–Blackwellized reverse-bridge objective and a corresponding stochastic reverse sampler.

• �� Introduce simplex-valued variables to enhance training objectives and reverse sampling processes.
• �� Retain the original categorical corruption process to ensure the integrity of the generative process.
• �� Achieve Rao–Blackwellized reverse-bridge objectives through simplex-valued variables.
• �� Conduct stochastic reverse sampling to improve generative performance.

Experiments were conducted on OpenWebText and Sudoku tasks to verify Simplax's performance. In the OpenWebText task, text generation was performed using the GPT-2 BPE tokenizer, evaluating generative perplexity and entropy. In the Sudoku task, models were trained using only 30-clue puzzles, evaluating solving accuracy across different clue densities and unconditional generation validity.

Experimental results show Simplax performs excellently in OpenWebText generation, especially in unconditional scenarios. In Sudoku tasks, Simplax excels across all evaluated clue densities, notably in the minimum uniquely solvable 17-clue regime. Experiments demonstrate Simplax performs well across different inference budgets, indicating its potential in various application scenarios.

Simplax can be used in scenarios such as text generation and Sudoku solving. In text generation, Simplax improves the perplexity-entropy tradeoff. In Sudoku solving, Simplax enhances solving accuracy and unconditional generation validity.

Simplax may perform poorly in certain complex scenarios, especially under extreme conditions. The method's reliance on simplex-valued variables may lead to increased computational overhead in some cases. Future work could explore Simplax's application in other generative tasks and investigate further optimization of simplex-valued variable usage to enhance model efficiency and performance.