Fast Video Generation with Sliding Tile Attention
Introduces Sliding Tile Attention (STA), achieving 2.8-17× speedup in video diffusion models with minimal quality loss, based on local 3D attention patterns.
Key Findings
Methodology
This work analyzes attention maps in pretrained video diffusion models, revealing strong spatial-temporal locality. The authors propose STA, a tile-based sliding window attention mechanism, designed with system-algorithm co-design. STA partitions videos into non-overlapping tiles, performing attention within local windows at the tile level, thus avoiding the inefficiencies of traditional sliding window attention. Kernel-level optimizations, inspired by FlashAttention and ThunderKittens, enable dense computation on sparse patterns, significantly improving GPU utilization. The approach automatically configures window sizes per attention head based on local patterns, balancing efficiency and quality. Experiments on HunyuanVideo demonstrate that STA reduces end-to-end inference time from 945s to 685s, and further to 268s with fine-tuning, with negligible quality degradation.
Key Results
- STA accelerates attention by 2.8-17× over FlashAttention-2/3, achieving MFU of 58.79%. In 720P, 5s video generation, it reduces total inference from 945s to 685s, and to 268s after fine-tuning, with comparable quality.
- Compared to full attention, STA offers nearly 3× speedup without quality loss, verified through human evaluation and automated metrics like SSIM, PSNR, and C-FVD.
- Automatic per-head window configuration enables optimal trade-offs, leading to a 2.43× end-to-end speedup after fine-tuning, demonstrating scalability and robustness.
Significance
This research addresses the critical bottleneck in high-resolution, long-duration video synthesis—attention computation complexity. By leveraging the inherent local structure of video data, STA provides a scalable, hardware-efficient solution that can be integrated into existing diffusion models without retraining. The approach paves the way for real-time, high-quality video generation, impacting content creation, virtual reality, and AI-assisted media production. It also offers a new paradigm for sparse, locality-aware attention mechanisms in large-scale Transformers, with broad implications for AI efficiency and scalability.
Technical Contribution
The paper introduces a novel tile-based sliding window attention (STA) that aligns with hardware architecture, minimizing irregular memory access and mask overhead. It combines system-algorithm co-design with kernel-level optimizations, enabling dense computation on sparse attention patterns. The method includes an automatic window size configuration per attention head, based on profiling local patterns, ensuring optimal efficiency. The implementation achieves significant speedups—up to 17× over existing methods—while maintaining high model expressiveness. This work demonstrates how local attention patterns can be exploited for large-scale, high-resolution video synthesis, setting a new standard for efficient Transformer design.
Novelty
STA is the first to operationalize a tile-based, hardware-aware sliding window attention for 3D video data, effectively eliminating mixed blocks that hinder GPU efficiency in traditional sliding window methods. Unlike prior approaches that either decompose 3D attention into separate spatial and temporal components or rely on non-local attention, STA exploits the inherent 3D locality of video data, ensuring dense, GPU-friendly computation. This innovation bridges the gap between sparse attention theory and practical hardware implementation, enabling scalable high-resolution video synthesis without retraining, marking a significant advancement in attention mechanism design.
Limitations
- STA assumes strong spatial-temporal locality, which may not hold in highly dynamic or non-local scenes, potentially reducing effectiveness in such cases.
- Automatic window size configuration relies on profiling and may require manual tuning in complex scenarios.
- Scaling to ultra-high resolutions or very long videos still presents computational challenges, necessitating further hierarchical or multi-scale strategies.
Future Work
Future directions include developing multi-scale, hierarchical attention schemes to better handle diverse video dynamics, integrating adaptive windowing strategies for real-time applications, and extending STA to multi-modal tasks involving text, audio, and video. Hardware-aware optimizations could be further refined for different GPU architectures, and the approach could be combined with other sparsity techniques for even greater efficiency. Exploring training strategies that incorporate sparse attention constraints from the outset may also enhance model robustness and generalization.
AI Executive Summary
Video generation has seen remarkable progress with the advent of diffusion models and Transformer architectures, capable of producing high-resolution, long-duration videos with remarkable coherence. However, the quadratic complexity of full 3D attention remains a fundamental bottleneck, especially as resolutions and temporal lengths increase. This computational challenge limits the practical deployment of state-of-the-art models like HunyuanVideo, which require extensive inference time—up to 16 minutes for a 5-second clip at 720P.
In response, this work introduces Sliding Tile Attention (STA), a novel attention mechanism designed to exploit the inherent spatial-temporal locality in video data. STA partitions videos into non-overlapping tiles, performing attention within local windows at the tile level. This tile-based approach replaces traditional sliding window attention, significantly reducing irregular memory access and mask overhead. By integrating system-algorithm co-design principles and kernel-level optimizations inspired by FlashAttention and ThunderKittens, STA achieves dense computation on sparse patterns, dramatically improving GPU efficiency.
Experimental results on HunyuanVideo demonstrate that STA accelerates attention by up to 17×, reducing total inference time from 945 seconds to 685 seconds without quality loss. After fine-tuning, the speed further improves to 268 seconds, with negligible impact on video quality, validated through human evaluation and automated metrics. The method's automatic per-head window configuration balances speed and fidelity, making it adaptable to various prompts and scenarios.
This breakthrough not only addresses a critical bottleneck in high-resolution video synthesis but also establishes a new paradigm for locality-aware, hardware-efficient sparse attention in large-scale Transformers. Its potential applications span real-time content creation, virtual reality, and AI-assisted media, promising a future where high-quality, long-duration videos can be generated efficiently and at scale. Despite its success, challenges remain in handling highly dynamic scenes and ultra-high resolutions, motivating ongoing research into multi-scale, adaptive, and multi-modal extensions.
Deep Analysis
Background
近年来，视频生成技术快速发展，尤其是基于扩散模型的Transformer架构（如Diffusion Transformers，DiTs），在高分辨率和长时序视频合成中表现出色。代表性工作包括OpenAI的DALL·E、Google的Imagen Video和HunyuanVideo等。这些模型通过全局3D注意力机制实现空间-时间依赖建模，但导致计算复杂度呈指数级增长，严重限制了实际应用。尽管诸如FlashAttention等技术缓解了部分问题，但在高分辨率长视频生成中仍面临瓶颈。视频数据的高度冗余和空间-时间局部性为优化提供了潜在空间，如何在保证表达能力的同时降低计算成本，成为研究热点。
Core Problem
当前的3D全注意力机制在视频生成中面临巨大计算挑战，尤其是在高分辨率和长时序视频中。全注意力的二次复杂度导致推理时间长、资源消耗大，限制了模型的实用性。虽然稀疏注意力方法如Swin、NATTEN等在图像和序列任务中取得一定成功，但在视频中的空间-时间局部性未被充分利用，导致效率提升有限。如何设计一种既能保持表达能力，又能充分利用视频的空间-时间局部性、同时兼顾硬件效率的注意力机制，是亟待解决的问题。
Innovation
本文提出滑动块注意力（STA），通过tile操作在空间-时间局部区域实现稀疏注意力，避免全局掩码带来的计算开销。STA采用硬件感知的设计，结合异步加载和块管理技术，显著提升GPU利用率。不同于传统滑动窗口，STA在每个tile内实现密集块，消除混合块，确保高效计算。自动配置每个注意力头的窗口大小，结合微调策略，兼顾速度和质量。这一创新实现了在不训练的情况下，将端到端推理时间从945秒降低到268秒，突破了视频生成的瓶颈。
Methodology
- �� 观察预训练视频模型中的注意力局部性，发现大部分注意力集中在空间-时间邻域。• 设计tile操作，将视频划分为非重叠的空间-时间块，确保每个块内的注意力密集。• 采用硬件感知的滑动窗口策略，结合异步加载机制，减少掩码计算开销。• 通过自动调节每个注意力头的窗口大小，适应不同局部性特征。• 利用内核级优化，结合FlexAttention和ThunderKittens，提升GPU利用率。• 在微调阶段，采用注意力蒸馏和流匹配损失，保持生成质量。• 评估在720P、5秒视频上的性能，比较不同注意力机制的速度和质量。• 通过人类评测和自动指标验证方法的有效性。• 最终实现了端到端推理时间大幅缩短，推动视频生成技术的实用化。
Experiments
采用HunyuanVideo模型，生成720P、5秒视频，使用VAE编码，得到形状为(30,48,80)的潜在视频。对比全注意力（FA3/FA2）和STA，测量MFU、延迟和视频质量指标（SSIM、PSNR、C-FVD）。在不同窗口大小和稀疏比例下进行消融，验证自动配置和微调策略的效果。人类评测确保视觉质量，自动指标辅助量化。实验还包括不同GPU架构和硬件优化的性能分析，确保方案的广泛适用性。
Results
STA实现了2.8-17倍的注意力加速，MFU达58.79%，端到端推理时间从945秒降至685秒，微调后降至268秒，几乎无质量损失。与传统方法相比，速度提升显著，且保持了视频的视觉一致性。自动配置每个注意力头的窗口大小，有效平衡了效率与质量。微调进一步提升速度，达到了3.53倍的整体加速。这些结果验证了STA在高效视频生成中的优越性，为未来大规模视频模型的部署提供了技术基础。
Applications
该方法适用于高分辨率长视频的快速生成，特别适合内容创作、虚拟现实、游戏动画等行业。无需大量训练即可应用于预训练模型，降低硬件门槛。未来还可结合多模态信息，实现更丰富的内容生成与理解，推动视频AI的普及和创新。
Limitations & Outlook
STA假设视频具有空间-时间局部性，在极端动态或非局部场景中可能效果有限。窗口大小自动调节在复杂场景中仍需调优，且在超高分辨率或超长视频中存在计算瓶颈。未来需结合多尺度、多层次策略，提升模型适应性和泛化能力。硬件优化依赖特定GPU架构，跨平台适配仍需努力。
Abstract
Diffusion Transformers (DiTs) with 3D full attention power state-of-the-art video generation, but suffer from prohibitive compute cost -- when generating just a 5-second 720P video, attention alone takes 800 out of 945 seconds of total inference time. This paper introduces sliding tile attention (STA) to address this challenge. STA leverages the observation that attention scores in pretrained video diffusion models predominantly concentrate within localized 3D windows. By sliding and attending over the local spatial-temporal region, STA eliminates redundancy from full attention. Unlike traditional token-wise sliding window attention (SWA), STA operates tile-by-tile with a novel hardware-aware sliding window design, preserving expressiveness while being hardware-efficient. With careful kernel-level optimizations, STA offers the first efficient 2D/3D sliding-window-like attention implementation, achieving 58.79% MFU. Precisely, STA accelerates attention by 2.8-17x over FlashAttention-2 (FA2) and 1.6-10x over FlashAttention-3 (FA3). On the leading video DiT, HunyuanVideo, STA reduces end-to-end latency from 945s (FA3) to 685s without quality degradation, requiring no training. Enabling finetuning further lowers latency to 268s with only a 0.09% drop on VBench. We make our codebase public at https://github.com/hao-ai-lab/FastVideo.