LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models
LLaDA MoE v2 scales mixture-of-experts diffusion models, trained on 23.5T tokens, nearing Qwen3 performance.
Key Findings
Methodology
The study employs Mixture-of-Experts Diffusion Language Models (MoE dLLM), systematically examining scaling behavior through optimization hyperparameters, compute allocation, and architecture design. Specifically, the optimal batch size grows faster with compute, while the learning rate decays more rapidly. IsoFLOP analysis shows the optimal token budget grows faster than activated model-side computation. Larger scales favor larger expert pools, and moderate expert granularity remains effective.
Key Results
- LLaDA MoE v2 trained on 23.5T tokens, using about 65% of Qwen3's pretraining tokens, approaches Qwen3 on several knowledge, reasoning, and coding benchmarks.
- After supervised fine-tuning, LLaDA MoE v2 outperforms SDAR Chat on seven reasoning and coding benchmarks.
- Experiments show larger expert pools and moderate expert granularity are effective across scales.
Significance
This research provides practical scaling laws and design principles for MoE dLLMs, filling a gap in the field. By systematically studying the scaling behavior of optimization hyperparameters, compute allocation, and architecture design, it offers crucial guidance for future large-scale language model design.
Technical Contribution
LLaDA MoE v2 demonstrates scaling trends distinct from existing autoregressive models, offering new theoretical guarantees and engineering possibilities. The study reveals unique advantages of MoE architecture in diffusion language models, particularly in compute allocation and architecture design.
Novelty
This study is the first to systematically examine the scaling behavior of MoE dLLMs, revealing trends distinct from autoregressive models. Its core innovation lies in uncovering unique scaling laws for optimization hyperparameters and compute allocation.
Limitations
- On some tasks, LLaDA MoE v2 still lags behind Qwen3, especially on Chinese knowledge benchmarks.
- The model's computational cost is high, potentially limiting practical applications.
Future Work
Future research could explore more efficient compute allocation strategies and architecture designs to further enhance model performance. Additionally, extending the study to more tasks and languages could verify its generality.
AI Executive Summary
LLaDA MoE v2 offers an alternative to autoregressive language models through Mixture-of-Experts Diffusion Language Models (MoE dLLM). The study systematically examines the scaling behavior of optimization hyperparameters, compute allocation, and architecture design, revealing trends distinct from autoregressive models.
In experiments, LLaDA MoE v2 is trained on 23.5T tokens, using about 65% of Qwen3's pretraining tokens, and approaches Qwen3 on several knowledge, reasoning, and coding benchmarks. After supervised fine-tuning, LLaDA MoE v2 outperforms SDAR Chat on seven reasoning and coding benchmarks.
This research provides practical scaling laws and design principles for MoE dLLMs, filling a gap in the field. Future research could explore more efficient compute allocation strategies and architecture designs to further enhance model performance.
Deep Analysis
Background
In recent years, large-scale language models have made significant advances in natural language processing. Autoregressive models generate text by predicting the next token sequentially, but their scalability is limited. Diffusion Language Models (dLLMs) offer a new generation method, decoding multiple tokens in parallel through an iterative denoising process. Mixture-of-Experts (MoE) models significantly enhance computational efficiency and scalability by introducing multiple expert modules.
Core Problem
While MoE models have succeeded in autoregressive language models, their scaling behavior in diffusion language models remains unclear. The study needs to systematically explore the role of optimization hyperparameters, compute allocation, and architecture design in diffusion language models to reveal unique scaling laws.
Innovation
The study is the first to systematically examine the scaling behavior of MoE dLLMs. By analyzing optimization hyperparameters, compute allocation, and architecture design, it reveals trends distinct from autoregressive models. These findings provide crucial guidance for future large-scale language model design.
Methodology
- �� Employ Mixture-of-Experts Diffusion Language Models (MoE dLLM)
- �� Systematically analyze scaling behavior of optimization hyperparameters, compute allocation, and architecture design
- �� Use IsoFLOP analysis to reveal optimal model-data allocation strategies
- �� Validate effectiveness of expert pools and granularity across scales through experiments
Experiments
The experiment trains LLaDA MoE v2 on 23.5T tokens, using about 65% of Qwen3's pretraining tokens. Benchmark tests include multiple knowledge, reasoning, and coding tasks. The model is compared with Qwen3 and SDAR Chat on reasoning and coding benchmarks to verify its performance.
Results
LLaDA MoE v2 approaches Qwen3 on several benchmarks, especially in reasoning and coding tasks. After supervised fine-tuning, the model outperforms SDAR Chat on seven reasoning and coding benchmarks. These results demonstrate the unique advantages of MoE architecture in diffusion language models.
Applications
MoE diffusion language models can be applied in various NLP fields, including automatic text generation, machine translation, and intelligent question-answering systems. Their efficient compute allocation strategies and architecture design provide significant advantages in large-scale data processing tasks.
Limitations & Outlook
Despite LLaDA MoE v2's excellent performance on many tasks, it still lags behind Qwen3 on some Chinese knowledge benchmarks. Additionally, the model's computational cost is high, potentially limiting its practical application. Future research could explore more efficient compute strategies to reduce costs.
Plain Language Accessible to non-experts
Imagine a large factory with many different experts, each responsible for different tasks. Traditional factories can only have one expert complete tasks one by one, while LLaDA MoE v2 is like a smart factory that can have multiple experts work together simultaneously. This collaboration greatly increases efficiency, allowing the factory to complete production tasks faster.
ELI14 Explained like you're 14
Imagine you're playing a massive multiplayer online game, where each player has their own skills. In traditional games, players need to complete tasks one by one, but LLaDA MoE v2 is like a super team that lets multiple players work together to complete tasks at the same time. This teamwork makes the game faster and more fun!
Glossary
Diffusion Language Model
A model that generates text through an iterative denoising process, allowing parallel decoding of multiple tokens.
Used in the paper as an alternative to autoregressive models.
Mixture-of-Experts
An architecture that enhances computational efficiency by introducing multiple expert modules.
Used to scale the computational capacity of language models.
IsoFLOP Analysis
A technique for analyzing model-data allocation strategies to determine optimal compute allocation.
Used to reveal trends in optimal token budget and model computation growth.
Autoregressive Model
A model that generates text by predicting the next token sequentially.
The traditional method compared with diffusion language models.
Supervised Fine-Tuning
A method of fine-tuning models with additional labeled data to improve performance on specific tasks.
Used to enhance LLaDA MoE v2's performance on reasoning and coding tasks.
Open Questions Unanswered questions from this research
- 1 How can model performance be further improved without increasing computational cost? Current methods still underperform on some tasks.
- 2 How do MoE diffusion language models perform on multilingual tasks? More research is needed to verify their generality.
Applications
Immediate Applications
Intelligent Question-Answering Systems
Utilize LLaDA MoE v2's efficient compute capabilities to develop smarter question-answering systems, enhancing user experience.
Long-term Vision
Automatic Text Generation
In the future, LLaDA MoE v2 could be used for large-scale automatic text generation, transforming the content creation industry.
Abstract
Diffusion language models (dLLMs) offer an alternative to autoregressive (AR) language modeling, yet the scaling behavior of Mixture-of-Experts (MoE) dLLMs remains poorly understood. We systematically characterize how optimization hyperparameters, compute allocation, and architecture scale for MoE dLLMs, identifying quantitative differences from scaling trends previously reported for AR models. Specifically, for optimization, the optimal nominal batch size grows faster, while the optimal learning rate decays more rapidly with compute. For model--data allocation, IsoFLOP analysis reveals a slight data-side tilt: the optimal token budget grows faster than activated model-side computation. For MoE architecture, larger scales increasingly favor larger expert pools at fixed activated capacity, while moderate expert granularity remains consistently effective and the preferred fraction of activated capacity assigned to shared experts remains stable across scales. Guided by these findings, we train LLaDA MoE v2, a 30B-A3B dLLM, from scratch on 23.5T tokens. With approximately 65\% as many pretraining tokens as Qwen3, LLaDA MoE v2 approaches Qwen3 on several knowledge, reasoning, and coding benchmarks. After supervised fine-tuning alone, it outperforms SDAR Chat on seven of eight reasoning and coding benchmarks and remains close to Qwen3 on several tasks. These results establish practical scaling laws and design principles for MoE dLLMs.