SGMD: Score Gradient Matching Distillation for Few-Step Video Diffusion Distillation
SGMD introduces gradient-based pseudo-score distillation with dual potentials, tripling training speed and enhancing motion dynamics in 4-step video diffusion.
Key Findings
Methodology
This paper proposes Score Gradient Matching Distillation (SGMD), which adopts a pseudo-score perspective, directly optimizing the fake score toward the teacher model, while employing teacher stop-gradient Fisher divergence as a stable distribution-matching objective. Theoretical analysis shows that under ideal tracking, this objective aligns with reverse KL divergence. SGMD introduces dual potentials: Negative Residual (NR) for outer-loop correction and Residual Contraction (RC) for inner-loop tracking, enabling low-overhead two-step optimization. During training, the generator is updated with LFisher and LNR, then the fake score with LRC, reducing the number of fake-score updates and accelerating training.
Key Results
- On the 14B Wan2.1-T2V-14B teacher, SGMD achieves approximately 3× training speedup in 4-step distillation, with FVD reduced to 100.3 (vs. 115.1 of DMD2), OptFlow improved to 9.29 (vs. 4.51), and significantly better motion dynamics and temporal consistency. Human evaluations favor SGMD in motion quality and overall preference, with 65% overall preference and 71% in motion quality. The model maintains comparable visual quality and text alignment, demonstrating superior dynamic richness and stability.
Significance
This work addresses the bottleneck of slow training and poor motion in few-step video diffusion distillation. By integrating gradient-based pseudo-score optimization with stable Fisher divergence, it enables faster training and richer motion, broadening practical applications in virtual reality, animation, and real-time content creation. The theoretical insights and empirical results provide a new paradigm for efficient, dynamic video synthesis, potentially transforming industry standards and enabling scalable, high-quality content generation.
Technical Contribution
The core innovation lies in combining pseudo-score gradient matching with teacher stop-gradient Fisher divergence, avoiding costly pseudo-score tracking. The dual potentials (NR/RC) facilitate low-cost, stable two-step updates, balancing distribution alignment and motion tracking. Theoretical analysis confirms the alignment of gradient directions with reverse KL divergence under ideal conditions. Empirically, SGMD outperforms existing methods in speed and motion richness, offering a new framework for efficient few-step distillation.
Novelty
This is the first systematic integration of pseudo-score gradient matching with teacher stop-gradient Fisher divergence, introducing dual potentials for low-overhead, stable training. Unlike prior reverse KL-based distillation, SGMD explicitly decouples correction and contraction, enabling faster convergence and richer dynamics. Its theoretical and empirical advantages mark a significant step forward in few-step video diffusion distillation.
Limitations
- The method still faces challenges in highly complex or fast-changing scenes, where tracking lag may persist. Further improvements are needed for extreme dynamic scenarios.
- Sensitivity to hyperparameter λ requires careful tuning, affecting robustness across different datasets and resolutions.
- Large-scale training remains computationally intensive; future work should explore more efficient training schemes and model compression.
Future Work
Future directions include adaptive mechanisms for tracking and distribution matching, multi-scale feature integration, and hardware acceleration to enable real-time high-resolution video synthesis. Extending SGMD to multi-modal content and exploring unsupervised or semi-supervised training paradigms could further broaden its impact.
AI Executive Summary
The rapid growth of video content demands efficient and high-quality generation methods. Diffusion models have demonstrated impressive results but are hampered by high computational costs, especially in multi-step sampling. Few-step distillation offers a promising solution, but existing approaches like Distribution Matching Distillation (DMD) face challenges in balancing training speed, motion richness, and temporal consistency. This paper introduces Score Gradient Matching Distillation (SGMD), a novel framework that leverages a pseudo-score perspective, optimizing the fake score directly toward the teacher model while employing a stable Fisher divergence as the distribution-matching objective. The key innovation is the integration of dual potentials—Negative Residual (NR) and Residual Contraction (RC)—which enable a low-overhead, two-step optimization process. Theoretical analysis confirms that, under ideal tracking, the gradient directions align with reverse KL divergence, ensuring stable and effective training.
Empirical results on large-scale video distillation with a 14B teacher model demonstrate that SGMD accelerates training by approximately three times compared to prior methods like DMD2. It significantly improves motion dynamics and temporal consistency, producing videos with richer, more vivid movements while maintaining high visual quality and text alignment. Human preference studies further validate the superiority of SGMD in motion quality and overall experience.
This work addresses fundamental bottlenecks in few-step video diffusion, offering a scalable, efficient, and dynamic content generation framework. Its theoretical foundations and practical performance suggest broad applicability in virtual reality, animation, and real-time video synthesis. Future research will focus on adaptive tracking, multi-scale features, and hardware acceleration to realize real-time, high-resolution video generation, pushing the boundaries of AI-driven content creation.
Deep Analysis
Background
近年来，视频生成技术迅速发展，扩散模型成为主流。代表性工作如Ho等的扩散采样、Wang等的高效架构，推动了高质量视频内容的生成。然而，模型计算成本高，限制了实际应用。为此，蒸馏技术被引入，尤其是少步蒸馏，通过减少采样步骤提升效率。尽管如此，现有方法在运动动态和时间一致性方面仍有不足，尤其在极少步数下追踪伪评分和保持复杂运动方面存在瓶颈。
Core Problem
核心问题在于少步蒸馏中伪评分追踪滞后和模式崩溃。逆KL匹配偏向保守，导致运动细节丧失。高频伪评分更新带来训练成本高、稳定性差。如何在提升速度的同时增强运动表现，成为亟待解决的难题。
Innovation
本文提出SGMD，创新点包括：1）采用伪评分梯度匹配，直接优化伪评分向教师模型靠近；2）利用教师停止梯度的Fisher散度，作为稳定的分布匹配目标，避免追踪滞后；3）引入双重势能（NR/RC），实现低成本双步优化，平衡追踪和匹配。理论分析显示在理想条件下，梯度方向与逆KL一致。实验证明，该方法在4步蒸馏中实现约3倍速度提升，运动动态明显优于现有方法。
Methodology
- �� 采用教师停止梯度的Fisher散度作为分布匹配目标，确保训练稳定。• 伪评分模型通过优化逼近教师评分，减少高成本追踪。• 引入双重势能：负残差（NR）用于外环校正，残差收缩（RC）用于内环追踪，通过对应损失实现。• 训练中，先用LFisher和LNR更新生成器参数，再用LRC优化伪评分模型，减少伪评分更新次数。• 采用两步交替优化策略，避免二阶导数计算，提升效率。
Experiments
在大规模视频蒸馏任务中，使用14B教师模型Wan2.1-T2V-14B，采用VBench、FVD和OptFlow指标评估。训练在32块H100 GPU上进行，使用AdamW，学习率1e-6。对比DMD2、TSG-Fisher和TSG-SIM，验证SGMD在4步蒸馏中的速度和运动表现。通过用户偏好和VideoAlign模型，验证运动质量和时间一致性优于对比方法，保持良好视觉和文本对齐。
Results
SGMD在4步蒸馏中实现约3倍训练加速，FVD降至100.3，OptFlow提升至9.29，运动丰富性和动态表现优于DMD2。用户偏好显示，65%的用户偏好SGMD，运动质量得分71%。模型在动态丰富性和运动表现方面显著优越，验证其实用性。
Applications
该技术适用于高效视频内容生成、虚拟现实、动画制作，特别适合快速生成动态丰富内容。结合硬件加速和多模态融合，有望实现实时高分辨率视频生成。
Limitations & Outlook
在极端复杂或高速场景中仍存在追踪滞后问题，超参数λ敏感，训练成本高，未来需优化追踪机制和参数自适应能力。
Plain Language Accessible to non-experts
想象你在做一份大餐，厨师需要不断调整火候和调料，确保每道菜都完美。传统方法就像用一个固定的食谱，可能会忽略菜肴的变化，导致味道不够丰富。现在，SGMD就像有一个智能助手，能实时观察厨师的操作，给出建议，让火候和调料都更合适。它通过不断学习厨师的手法，快速调整，节省时间又保证菜肴的美味。这个助手用一种特别的方式，确保每次调整都更准确，不会偏离目标。这样，厨师可以更快做出色香味俱佳的菜肴，整个过程既高效又有趣。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，你的目标是让角色跑得更快、跳得更高，但每次你调整动作时都要花很多时间观察和试错。传统的方法就像用一个固定的训练计划，虽然能帮你变强，但速度慢，效果也不一定好。现在，SGMD就像有个聪明的教练，他会观察你每次的动作，给你一些建议，让你更快找到正确的跑跳姿势。这个教练不会一直盯着你，而是用一种特别的方法，确保每次建议都很靠谱，不会让你偏离目标。这样，你就能用更少的练习时间，变得更厉害，游戏体验也更棒。
Glossary
Score Gradient Matching (评分梯度匹配)
一种通过优化伪评分的梯度，使其逐步逼近教师模型评分的方法，旨在提升生成模型的训练效率和动态表现。
本文中用以指导伪评分模型向教师评分靠近，改善少步蒸馏中的追踪问题。
Fisher Divergence (Fisher散度)
衡量两个概率分布之间差异的指标，利用梯度信息进行匹配，具有平滑性和稳定性。
作为稳定的分布匹配目标，避免伪评分追踪滞后带来的训练不稳定。
伪评分 (Fake Score)
由辅助网络生成的估计目标，用于引导生成模型训练，模拟真实评分。
在蒸馏过程中用以替代昂贵的真实评分，提升训练效率。
双重势能 (Dual Potentials)
引入两个相反作用的损失函数，用于平衡追踪和校正过程。
在SGMD中，分别为负残差（NR）和残差收缩（RC），实现低成本的双步优化。
Open Questions Unanswered questions from this research
- 1 如何在极端复杂或高分辨率视频中进一步优化追踪机制，减少滞后，提高运动表现，是未来的关键问题。
- 2 目前方法对超参数λ敏感，缺乏自适应调节机制，影响模型在不同场景下的泛化能力。
- 3 大规模训练成本仍较高，需探索更高效的训练策略和模型压缩技术，以实现更广泛的应用。
Applications
Immediate Applications
高效视频内容生成
利用SGMD快速训练少步视频扩散模型，满足虚拟现实、动画制作等行业对高质量动态内容的需求，降低硬件成本，提升生产效率。
虚拟现实与增强现实
实现实时动态视频生成，增强沉浸感和交互体验，为虚拟场景和游戏开发提供技术支撑。
Long-term Vision
智能内容创作平台
结合SGMD实现大规模、多模态视频生成，推动自动化动画、电影制作等产业变革，缩短创作周期。
Abstract
Distribution Matching Distillation (DMD) is a widely used paradigm for accelerating inference in few-step video diffusion models. However, DMD-style video distillation faces two coupled challenges: the fake score must track a continuously evolving generator, making training costly when frequent updates are required, while reverse-KL-style matching can be mode-seeking and conservative for preserving strong motion dynamics. To address these issues, we propose \textbf{Score Gradient Matching Distillation (SGMD)}. SGMD adopts a fake-score perspective by directly optimizing the fake score toward the teacher, while using teacher stop-gradient Fisher as a stable distribution-matching objective. We provide a gradient analysis that motivates this objective choice under ideal tracking. Building on this, SGMD introduces a pair of dual potentials: negative-residual (NR) for outer-loop correction and residual-contraction (RC) for inner-loop tracking. Empirically, compared to DMD2, SGMD achieves an approximately $\sim 3\times$ training speedup and substantially improves motion dynamics for 4-step distilled models while preserving temporal consistency. A human study confirms that SGMD is preferred in motion quality and overall preference, while visual quality and text alignment remain comparable. Code is available at https://github.com/ModelTC/LightX2V.