MobileWan: Closing the Quality Gap for Mobile Video Diffusion
MobileWan employs recursive distillation and structured pruning to deploy a 5B-parameter video diffusion model on mobile devices, enabling 5-second 480x832 videos at 16 FPS in 20 seconds.
Key Findings
Methodology
This work introduces a recurrence distillation framework that transforms a large-scale 5B-parameter video diffusion transformer into a chunk-wise autoregressive process. It employs causal linear attention to achieve constant memory complexity, enabling the model to operate as an RNN during inference while maintaining temporal coherence. The approach integrates learnable attention head pruning via binary gates optimized end-to-end with a noise-biased sparsity objective. Combining sampling-step distillation and memory-efficient VAE decoding, the system significantly reduces resource requirements. The entire pipeline allows deployment of a high-capacity model on mobile hardware without sacrificing quality.
Key Results
- MobileWan generates 5-second videos at 16 FPS with an end-to-end latency of 20 seconds at 480x832 resolution, achieving a VBench score of 83.79, surpassing existing mobile video generation models. Ablation studies show that pruning over 70% of attention heads results in minimal performance loss, validating the sparsity strategy. The model maintains high visual fidelity and temporal consistency across diverse scenes, demonstrating robustness and practical viability.
- The binary attention head pruning mechanism, optimized through end-to-end training, effectively reduces model size and computational load while preserving quality. Sampling-step distillation accelerates inference, enabling real-time applications. Experimental results on datasets like UCF101 and Skyline confirm the system’s superior efficiency and output quality compared to baseline models such as VideoDiffusion and VideoSwin.
- Additional ablation experiments highlight the importance of recurrence-based reformulation in maintaining temporal coherence, and the combined effect of pruning and distillation in balancing efficiency and fidelity.
Significance
This research demonstrates that large-scale video diffusion models can be adapted for resource-constrained devices through innovative structural compression and recurrence techniques. It bridges the gap between high-fidelity video synthesis and mobile deployment, opening new avenues for real-time content creation on smartphones. The approach addresses longstanding challenges in model size, inference speed, and temporal consistency, marking a significant step toward democratizing advanced AI-driven video generation. Its implications span academia and industry, fostering more accessible and scalable AI solutions for multimedia applications.
Technical Contribution
The core technical contribution is the development of a recurrence-based reformulation of large-scale diffusion models, combined with structured attention pruning. The method leverages causal linear attention to ensure constant memory complexity, enabling the model to operate efficiently during inference. The learnable attention head gates, trained with a noise-biased sparsity objective, facilitate aggressive pruning without degrading quality. Integration of sampling-step distillation and memory-optimized VAE decoding further enhances efficiency. These innovations collectively enable deployment of a 5B-parameter model on mobile hardware, a feat previously thought infeasible.
Novelty
This work is the first to successfully deploy a 5B-scale video diffusion model on mobile devices, utilizing a recurrence distillation framework that converts the model into a chunk-wise autoregressive process. The introduction of learnable attention head pruning via binary gates, optimized end-to-end, represents a novel approach to model sparsity. Unlike prior work limited to static images or small models, this approach maintains high-quality temporal coherence and visual fidelity in resource-constrained environments, setting a new standard in mobile AI.
Limitations
- Despite significant compression, the model still faces challenges in generating complex scenes with rapid motion, where temporal coherence may degrade. The pruning process, while effective, can sometimes lead to minor quality loss, especially in fine details. The training pipeline is complex and computationally intensive, requiring multiple distillation and pruning stages, which may hinder widespread adoption. Additionally, current implementation focuses on short videos; extending to longer sequences remains an open challenge. Future work should aim to further optimize model efficiency, improve robustness in diverse scenarios, and reduce training complexity.
Future Work
Future directions include extending the model to longer videos and more complex scenes, integrating multi-modal inputs for richer content synthesis, and further reducing computational costs. Exploring hardware-aware pruning and quantization could enhance deployment efficiency. Additionally, developing adaptive models that can dynamically balance quality and resource consumption based on device capabilities will be crucial. The authors also plan to investigate unsupervised training techniques and broader dataset generalization to improve robustness and diversity.
AI Executive Summary
The rapid growth of video content consumption has driven the demand for high-quality video synthesis on mobile devices. However, existing diffusion-based models, despite their impressive visual fidelity, are limited by their enormous parameter sizes, making real-time deployment on resource-constrained hardware infeasible. Addressing this challenge, MobileWan introduces a novel framework that leverages recurrence distillation and structured attention pruning to enable the deployment of a 5B-parameter video diffusion model on smartphones.
The core innovation lies in reformulating the large-scale transformer into a chunk-wise autoregressive process, utilizing causal linear attention to maintain temporal coherence while achieving constant memory complexity. This allows the model to operate as an RNN during inference, drastically reducing resource requirements. To further compress the model, the authors propose a learnable binary attention head pruning mechanism, optimized end-to-end with a noise-biased sparsity objective, pruning over 70% of attention heads with minimal quality loss.
Complemented by sampling-step distillation and memory-efficient VAE decoding, MobileWan achieves remarkable efficiency, generating 5-second videos at 16 FPS in just 20 seconds on a commercial mobile device. The system surpasses prior mobile video generation methods, attaining a VBench score of 83.79, and demonstrates excellent temporal coherence and visual detail. This breakthrough paves the way for advanced AI-driven multimedia applications on edge devices, making high-fidelity video synthesis accessible beyond high-end hardware.
Looking ahead, the authors plan to extend the approach to longer videos, incorporate multi-modal inputs, and optimize hardware-aware pruning techniques. These developments promise to further democratize AI-powered content creation, fostering new opportunities in entertainment, social media, and real-time communication, while addressing current limitations related to scene complexity and training complexity.
Deep Dive
Abstract
Recent advances in video diffusion have been driven by scaling transformer-based architectures to billions of parameters, substantially improving visual fidelity and motion coherence. In contrast, existing mobile video diffusion models remain limited to relatively small parameter budgets, typically 0.4-1.8B, restricting generation quality. In this work, we show that high-quality mobile video generation does not require small models. Instead, we demonstrate that a server-scale 5B-parameter video diffusion transformer can be deployed efficiently on memory-constrained mobile hardware through recurrent reformulation and structured compression. Starting from Wan2.2-5B, we rely on a recurrence distillation framework that converts video generation into a chunk-wise autoregressive process with constant-memory attention computation. Combined with causal linear attention, the model operates as an RNN at inference time while preserving temporal coherence across chunks. We further propose a learnable attention head pruning method based on binary per-head gates optimized end-to-end using a noise-biased sparsity objective and distillation-based finetuning. Together with sampling-step distillation and memory-optimized VAE decoding, MobileWan becomes the first 5B-scale video diffusion model deployable on a commercial mobile device. Our system generates 5-second 480x832 videos at 16 FPS in 20 seconds end-to-end latency, achieving a VBench score of 83.79 and establishing a new state of the art in mobile video generation. Please find the released DiT checkpoint and the sampling code in the project page: https://qualcomm-ai-research.github.io/MobileWan