DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps
DPM-Solver uses analytical diffusion ODE solutions, enabling high-quality sampling in 10-20 steps, with 4-16× speedup over prior methods.
Key Findings
Methodology
This work derives an exact solution formula for diffusion ODEs by analytically computing the linear part, transforming the problem into an exponentially weighted neural network integral. Leveraging the semi-linear structure, the authors design high-order DPM-Solvers (1st to 3rd order) with convergence guarantees. The approach avoids black-box discretization errors, enabling fewer steps for high-quality sampling. The method supports both discrete and continuous DPMs without retraining, using change-of-variable techniques and exponential integrator principles to efficiently approximate the integral. Adaptive step size strategies further enhance practical performance.
Key Results
- On CIFAR-10, DPM-Solver achieves an FID of 4.70 with only 10 function evaluations, and 2.87 with 20 evaluations, outperforming prior training-free methods by a large margin. The speedup ranges from 4× to 16× while maintaining high sample quality.
- In ImageNet 256×256, the method produces high-quality samples rapidly, with fewer steps. It generalizes well across datasets, noise schedules, and model types, demonstrating robustness.
- Compared to general-purpose ODE solvers like RK45, DPM-Solver exhibits smaller discretization errors and greater stability in few-step regimes, validating the advantage of exploiting the semi-linear structure.
Significance
This approach fundamentally advances the efficiency of diffusion-based generative models, enabling high-quality sampling with minimal steps without additional training. It addresses a critical bottleneck, making DPMs more practical for real-world applications such as real-time image synthesis and large-scale deployment. The theoretical insights into the structure of diffusion ODEs open new avenues for algorithm design, bridging the gap between mathematical analysis and practical acceleration.
Technical Contribution
The main contribution is the derivation of an exact analytical solution for semi-linear diffusion ODEs, transforming the solution into an exponential integral form. This enables the construction of high-order, stable, and efficient solvers that outperform traditional black-box methods. The work introduces a novel perspective on leveraging the structure of the ODEs, guaranteeing convergence order, and integrating adaptive step size control, thus significantly improving sampling speed and quality.
Novelty
This is the first work to explicitly derive and utilize the analytical solution structure of diffusion ODEs, revealing that the solution decomposes into a linear part and an exponential integral of the neural network. Unlike prior black-box numerical methods, this approach exploits the semi-linear form for precise, fast approximation, enabling high-order, few-step sampling without training. It fundamentally redefines how diffusion models can be efficiently sampled.
Limitations
- The method relies on the semi-linear structure; its applicability to non-linear or more complex models remains limited. Extending the approach to broader classes of ODEs requires further research.
- Higher-order solvers (beyond third order) demand more intermediate points, increasing computational complexity. Optimization is needed for large-scale deployment.
- Sensitivity to noise schedule parameters and model hyperparameters may affect stability and performance, requiring careful tuning across tasks.
Future Work
Future directions include generalizing the analytical solution to more complex ODE forms, integrating learning-based step size adaptation, and extending to multi-modal and conditional generation tasks. Combining this approach with model fine-tuning or distillation could further accelerate sampling, broadening practical impact.
AI Executive Summary
Diffusion probabilistic models (DPMs) have revolutionized generative tasks such as image synthesis, but their slow sampling process remains a major obstacle. Traditional methods require hundreds or thousands of neural network evaluations, limiting real-time applications. This paper introduces DPM-Solver, a novel numerical approach grounded in the analytical structure of diffusion ODEs. By precisely computing the linear component and transforming the solution into an exponential integral, the authors develop high-order solvers that guarantee convergence and drastically reduce the number of steps needed.
The core innovation lies in exploiting the semi-linear form of diffusion ODEs, enabling the derivation of an exact solution formula. This formula simplifies the problem into approximating an exponential integral of the neural network, which can be efficiently handled using exponential integrator techniques. The resulting solvers (1st to 3rd order) are capable of producing high-quality samples with as few as 10-20 function evaluations, a significant improvement over existing approaches.
Experimental results across datasets like CIFAR-10 and ImageNet demonstrate that DPM-Solver achieves state-of-the-art speed and quality, with FID scores of 4.70 and 2.87 respectively, while maintaining stability and robustness. Compared to general-purpose ODE solvers such as RK45, DPM-Solver exhibits smaller discretization errors and better stability in few-step regimes. The method's flexibility allows it to support both discrete and continuous-time models without retraining, making it highly practical.
This work offers a new theoretical perspective on the structure of diffusion ODEs, opening pathways for further acceleration and broader applicability. Despite some limitations in extending beyond semi-linear forms, it marks a significant step toward making diffusion models more efficient and accessible for real-world deployment, especially in scenarios demanding rapid, high-quality generation.
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房做一道复杂的菜肴。传统做法像是每次都用慢慢试错的方法，一点点调料，耗费大量时间。而现在，你学会了一种新技巧：提前把一些关键步骤和调料用特殊的方法预处理，只需少量操作就能做出美味佳肴。这就像论文中的DPM-Solver，利用数学公式提前算出一部分内容，把复杂的步骤拆解成简单的部分，快速组合，节省时间又保证味道。它用一种聪明的办法，把复杂的烹饪变成几次简单的操作，既快又好吃。
ELI14 Explained like you're 14
你知道在游戏里打boss，打得越快越好吗？以前用普通武器，要打很多次才能赢，耗时长。而这个新方法就像用超级技能，一下子就把boss打倒，只用几次攻击！它的秘密在于提前算出一些关键步骤，把复杂的战斗拆成几招，然后快速组合。这样，不仅节省时间，还能打得更漂亮。这篇论文的技巧就像用数学魔法，把复杂的战斗变得简单，用少量操作就能赢，真是太厉害了！
Abstract
Diffusion probabilistic models (DPMs) are emerging powerful generative models. Despite their high-quality generation performance, DPMs still suffer from their slow sampling as they generally need hundreds or thousands of sequential function evaluations (steps) of large neural networks to draw a sample. Sampling from DPMs can be viewed alternatively as solving the corresponding diffusion ordinary differential equations (ODEs). In this work, we propose an exact formulation of the solution of diffusion ODEs. The formulation analytically computes the linear part of the solution, rather than leaving all terms to black-box ODE solvers as adopted in previous works. By applying change-of-variable, the solution can be equivalently simplified to an exponentially weighted integral of the neural network. Based on our formulation, we propose DPM-Solver, a fast dedicated high-order solver for diffusion ODEs with the convergence order guarantee. DPM-Solver is suitable for both discrete-time and continuous-time DPMs without any further training. Experimental results show that DPM-Solver can generate high-quality samples in only 10 to 20 function evaluations on various datasets. We achieve 4.70 FID in 10 function evaluations and 2.87 FID in 20 function evaluations on the CIFAR10 dataset, and a $4\sim 16\times$ speedup compared with previous state-of-the-art training-free samplers on various datasets.