Few-Step Diffusion Language Models via Trajectory Self-Distillation
Few-Step Diffusion Language Models via Trajectory Self-Distillation improve reasoning tasks.
Key Findings
Methodology
The study proposes a trajectory self-distillation framework that trains a few-step student model by matching the generative trajectory of a full-step teacher model. This method incorporates Direct Discriminative Optimization (DDO), a reverse-KL objective to encourage mode-seeking towards the teacher's high-probability modes, thereby enhancing performance on reasoning tasks.
Key Results
- On reasoning and code-generation benchmarks, the gap between few-step and full-step decoding is significantly narrowed, with accuracy on the MATH500 dataset improved to 60%.
- T3D outperforms existing methods across multiple decoding budgets on SDAR and LLaDA models, especially under aggressive budgets.
- DDO helps maintain high accuracy in reasoning tasks by significantly reducing conditional total correlation.
Significance
This research is significant for both academia and industry, addressing the inefficiency of large-scale language model inference. By reducing decoding steps without sacrificing generation quality, T3D offers a more efficient solution for real-time and compute-constrained applications.
Technical Contribution
T3D introduces trajectory self-distillation and DDO to significantly reduce factorization error in few-step decoding, providing richer supervision than existing endpoint-based methods. The method is theoretically proven to reduce conditional total correlation.
Novelty
This is the first application of trajectory self-distillation in discrete diffusion language models, differing from previous endpoint supervision methods by effectively reducing factorization error through intermediate state supervision.
Limitations
- In some complex reasoning tasks, few-step models may still experience a decline in generation quality.
- The model's performance under dynamic decoding remains to be further validated.
Future Work
Future research could explore applications on more tasks and model architectures, as well as further optimize the efficiency of trajectory self-distillation.
AI Executive Summary
Diffusion large language models (DLLMs) have emerged as powerful generative models for text generation, yet their inference efficiency is limited by the number of decoding steps. Reducing these steps often leads to a significant drop in output quality. To address this, researchers propose a trajectory self-distillation framework that trains a few-step student model by matching the generative trajectory of a full-step teacher model. This method incorporates Direct Discriminative Optimization (DDO), a reverse-KL objective that encourages the student model to focus on the teacher's high-probability modes, thereby enhancing performance on reasoning tasks.
On reasoning and code-generation benchmarks, the T3D method significantly narrows the gap between few-step and full-step decoding, with accuracy on the MATH500 dataset improved to 60%. By using DDO, the few-step model maintains high accuracy in reasoning tasks, significantly reducing conditional total correlation. This research is significant for both academia and industry, addressing the inefficiency of large-scale language model inference.
While T3D maintains generation quality while reducing decoding steps, few-step models may still experience a decline in generation quality in some complex reasoning tasks. Additionally, the model's performance under dynamic decoding remains to be further validated. Future research could explore applications on more tasks and model architectures, as well as further optimize the efficiency of trajectory self-distillation.
Deep Analysis
Background
Diffusion large language models (DLLMs) have recently made significant advances in text generation. However, these models typically require long decoding chains, limiting their efficiency in real-time and compute-constrained applications. Existing research mainly focuses on system and decoding-level improvements, such as better decoding strategies and adaptive KV caching.
Core Problem
Reducing the number of decoding steps often leads to a significant drop in output quality due to increased factorization error in few-step decoding. Existing endpoint supervision methods fail to fully utilize the generative trajectory information of the teacher model.
Innovation
Trajectory self-distillation trains a few-step student model by matching the generative trajectory of a full-step teacher model, combined with Direct Discriminative Optimization (DDO) to enhance performance on reasoning tasks. Compared to previous endpoint supervision methods, this approach provides richer supervision.
Methodology
- �� Trajectory Self-Distillation: Trains a few-step student model by matching the teacher model's generative trajectory.
- �� Direct Discriminative Optimization (DDO): Encourages the student model to focus on the teacher's high-probability modes.
- �� Path-Consistency Regularization: Assigns higher training weights to tokens decoded earlier in the trajectory.
Experiments
Experiments were conducted on benchmarks such as MATH500, GSM8K, MBPP, and HumanEval using SDAR and LLaDA models. Evaluation metrics include accuracy and conditional total correlation.
Results
T3D outperforms existing methods across multiple decoding budgets, especially under aggressive budgets. Accuracy on the MATH500 dataset improved to 60%.
Applications
T3D can be applied in real-time text generation and compute-constrained scenarios, such as intelligent assistants and automated code generation.
Limitations & Outlook
While T3D maintains generation quality while reducing decoding steps, few-step models may still experience a decline in generation quality in some complex reasoning tasks. Additionally, the model's performance under dynamic decoding remains to be further validated.
Plain Language Accessible to non-experts
Imagine you're in a kitchen, where you usually prepare ingredients step by step and then cook according to a recipe. Sometimes, you want to finish the dish faster, so you decide to do multiple steps at once, like chopping vegetables while boiling soup. The challenge is to ensure each step is accurate, or the final dish might lose its original flavor. Trajectory self-distillation is like a smart assistant that helps you maintain accuracy in each step while doing multiple steps simultaneously, ensuring the final dish is delicious.
ELI14 Explained like you're 14
Imagine you're playing a game where you usually complete tasks step by step, but sometimes you want to finish faster. So, you decide to complete multiple tasks at once to save time. The risk is that if one task goes wrong, it might affect the entire game's progress. Trajectory self-distillation is like a game guide that helps you maintain accuracy in each task while completing multiple tasks simultaneously, ensuring you can smoothly pass the level. Isn't that cool?
Glossary
Diffusion Model
A generative model that generates data by gradually adding noise.
Used as the foundational model for generating text sequences.
Self-Distillation
A training method that uses the output of a teacher model to guide the learning of a student model.
Used to reduce factorization error in few-step decoding.
Direct Discriminative Optimization
An optimization objective that encourages the model to focus on high-probability modes.
Used to improve the accuracy of few-step decoding.
Conditional Total Correlation
A metric that measures factorization error during decoding.
Used to evaluate the quality of few-step decoding.
Path-Consistency Regularization
A regularization method that assigns higher training weights to tokens decoded earlier.
Used to reduce error propagation in few-step decoding.
Open Questions Unanswered questions from this research
- 1 How can the performance of few-step models be further improved under dynamic decoding? Current methods mainly target static decoding.
- 2 How to maintain generation quality in more complex reasoning tasks for few-step models?
- 3 How to further reduce factorization error without increasing computational cost?
Applications
Immediate Applications
Intelligent Assistants
Enhance response speed and accuracy in intelligent assistants through more efficient text generation.
Automated Code Generation
Improve generation efficiency in code generation tasks by reducing decoding steps.
Long-term Vision
Real-Time Translation Systems
Achieve faster real-time translation by reducing decoding steps.
Abstract
Diffusion large language models (DLLMs) have emerged as powerful generative models with the promise of fast text generation through parallel decoding. However, realizing this potential in practice remains challenging: reducing the number of decoding steps, typically causes a substantial degradation in output quality due to token factorization error. To alleviate this, we propose a self-distillation framework that trains a few-step student to match the generative trajectory of a full-step teacher. We theoretically and empirically show that trajectory-level supervision mitigates this factorization error, thereby enabling effective few-step decoding. We further incorporate Direct Discriminative Optimization (DDO), a reverse-KL objective that encourages mode-seeking toward the teacher's modes, yielding stronger performance on challenging reasoning tasks. Across reasoning and code-generation benchmarks, our method substantially narrows the gap between few-step and full-step decoding. The source code is available at https://github.com/Tyrion58/T3D.