How Far Can 5,500 Hours of Driving Take You? A Scaling Law Analysis of Video Diffusion Models
Scaling laws for video diffusion models in driving data show validation loss follows power laws; a 9B-parameter model predicts loss of ~0.0753 with 3.6% error.
Key Findings
Methodology
Using a power-law framework, the study analyzes how validation loss depends on model size, training exposure, and compute. Over 200 experiments across models from 1M to 9B parameters, with varied training data and compute, fit a multi-dimensional scaling law. The architecture employs Diffusion Transformer with Flow Matching, trained on a large-scale driving dataset of 5500 hours, covering 28 countries. The analysis involves fitting asymptotic power laws along three axes and extrapolating to predict the performance of a 9B-parameter model, validated through extensive experiments. The approach combines empirical data fitting with theoretical modeling to understand the scaling behavior in data-constrained environments.
Key Results
- Validation loss decreases with model size following a power law with an exponent αN≈0.213, from 0.1528 at 1.6M parameters to 0.0828 at 1.1B parameters, predicting ~0.0753 for 9B parameters with only 3.6% error.
- Training exposure impacts performance more strongly, with αD≈0.74, showing that longer training significantly reduces validation loss across all model sizes, reaching near-optimal performance within moderate exposure (~28M samples).
- Compute scaling reveals validation loss follows a power law with αC≈0.155, with the largest model (9B) reaching predicted performance (~0.0753) after approximately 1.4×10^7 samples, confirming the validity of the scaling law across resources.
Significance
This work provides the first systematic validation of scaling laws in video diffusion models for driving scenes, offering a quantitative basis for optimizing model size, training duration, and compute. It addresses a critical bottleneck in data-limited autonomous driving applications, guiding efficient resource allocation. The findings demonstrate that longer training surpasses model scaling in performance gains under fixed data, informing future model development and deployment strategies. The ability to predict performance based on these laws accelerates the design of large-scale, high-fidelity virtual driving environments, crucial for safety validation and simulation-based training.
Technical Contribution
The paper introduces a multi-dimensional power-law model capturing the dependence of validation loss on model size, training exposure, and compute. It extends prior scaling law frameworks from language models to video diffusion, incorporating Flow Matching and Transformer architectures. The empirical validation across extensive experiments establishes the predictive accuracy of these laws, enabling reliable extrapolation to larger models. The work also demonstrates training a 9B-parameter diffusion model from scratch, setting new benchmarks for driving video generation, and providing a scalable framework for future research in data-constrained regimes.
Novelty
This is the first comprehensive study establishing scaling laws for video diffusion models trained on real driving data, specifically under data-limited conditions. It combines empirical fitting with theoretical modeling to predict performance beyond current training scales, filling a gap in understanding the scalability of generative video models. The integration of Flow Matching with Transformer architectures in a large-scale setting and the successful training of a 9B-parameter model from scratch represent significant innovations, advancing the field of autonomous driving simulation.
Limitations
- Despite promising results, training such large models requires immense computational resources, limiting accessibility and practical deployment.
- The dataset, though large, is still finite, and the models' generalization to unseen, complex driving scenarios remains unverified.
- Model inference speed and real-time deployment in autonomous systems pose challenges, necessitating further optimization.
Future Work
Future directions include integrating multimodal data (depth, object detection) to enhance scene understanding, exploring model compression for real-time inference, and expanding datasets to cover more diverse driving environments. Additionally, investigating transfer learning and domain adaptation could improve generalization. Further research on scaling laws in multi-modal and multi-task settings will refine resource allocation strategies, accelerating the deployment of large-scale generative models in real-world autonomous systems.
AI Executive Summary
This study systematically investigates the scaling behavior of video diffusion models trained on autonomous driving data, revealing that validation loss follows predictable power-law relationships across model size, training exposure, and compute. By extensive experimentation, the authors demonstrate that increasing model capacity yields lower asymptotic loss, but training duration has a more significant impact on performance improvements within data-constrained environments. The empirical scaling laws enable accurate prediction of a 9B-parameter model's performance, achieving a validation loss of approximately 0.0753 with only 3.6% deviation from predictions. This model, trained from scratch, sets a new state-of-the-art benchmark for driving video generation, validated on nuScenes dataset. The findings guide optimal resource allocation, emphasizing longer training over larger models when compute is limited, while also confirming that larger models continue to benefit from additional compute and data. The work bridges the gap between theoretical scaling laws and practical large-scale model training, providing a scientific basis for future development of high-fidelity, data-efficient autonomous driving simulators. Despite computational challenges, these results mark a significant step toward scalable, controllable, and high-quality virtual driving environments, essential for safety validation and system robustness in autonomous vehicles.
Deep Analysis
Background
随着深度学习在视觉和生成任务中的突破，视频生成技术逐渐成熟，尤其在自动驾驶领域，虚拟场景的高质量合成成为研究热点。早期方法主要依赖GAN和自回归模型，但存在训练不稳定、难以扩展的问题。近年来，扩散模型凭借其稳定性和优异的生成质量逐渐占据主导地位，代表性工作如DALL·E、Imagen等推动了图像生成的边界。然而，视频生成面临更高的复杂度和数据需求，尤其在自动驾驶中，真实场景多样且复杂，数据采集成本高，隐私限制严格。现有研究多集中在有限数据集或单一模型，缺乏系统的规模效应分析。本研究旨在填补这一空白，系统验证在有限驾驶数据环境下，视频扩散模型的规模定律，为未来大规模训练提供理论基础。
Core Problem
在自动驾驶场景中，虚拟环境的高质量生成依赖于大规模模型和丰富数据，但实际数据采集成本高、隐私限制严苛，导致数据资源有限。如何在有限数据和计算资源条件下，最大化模型性能成为核心难题。现有模型多受限于训练时间、模型容量和数据重复率，缺乏对规模效应的系统理解。尤其在驾驶场景中，数据的多样性和复杂性要求模型具有强泛化能力。如何合理配置模型参数、训练时间和数据利用率，确保模型在有限数据条件下达到最优性能，成为亟待解决的问题。这关系到模型的实用性和安全性，影响自动驾驶系统的可靠性。
Innovation
本研究提出了多维规模定律模型，系统分析模型参数、训练曝光和计算资源对验证损失的影响。创新点包括：
- �� 采用幂律关系拟合验证损失，揭示不同规模和训练时间的性能变化规律；
- �� 在驾驶场景下训练最大规模（9B参数）视频扩散模型，验证模型性能的可预测性；
- �� 结合Flow Matching机制，提升模型训练的稳定性和生成质量；
- �� 通过大规模实验，验证模型在有限数据环境中的规模效应，提供科学的模型扩展策略。此方法突破了以往仅关注单一因素的研究，为多因素协同优化提供了理论基础。
Methodology
- �� 构建多维幂律模型，分析模型参数（N）、训练曝光（D）和计算预算（C）对验证损失（L）的影响；
- �� 采用Diffusion Transformer架构，结合Flow Matching训练机制，利用多摄像头驾驶数据集进行训练；
- �� 设计超过200次训练实验，涵盖不同模型规模（1.6M到9B参数）、训练样本（最多2.8亿）和计算资源，拟合验证损失的幂律关系式；
- �� 通过模型参数、训练曝光和计算预算的幂律拟合，预测最大模型的性能极限，验证预测准确性；
- �� 在最大模型（9B参数）上进行长时间训练，验证模型性能与预测的一致性，确保模型在实际应用中的可扩展性。
Experiments
实验使用由NATIX提供的5500小时多摄像头驾驶视频，划分为约30万段1分钟视频片段。模型在不同参数规模（从1.6M到9B参数）上训练，采用不同学习率和批次大小，评估验证损失。通过拟合验证损失的幂律关系，分析模型参数、训练样本数和计算资源的影响。还进行了数据重复率的消融实验，验证有限数据下的性能极限。所有模型均在GPU集群上训练，采用bfloat16精度，利用AdamW优化器，训练时间从几小时到数天不等。最终，基于拟合模型的预测，训练了最大规模（9B参数）模型，并验证其性能与预测的一致性。
Results
验证损失随模型参数呈幂律递减，预测最大模型（9B参数）验证损失约为0.0753，误差仅3.6%。训练曝光对性能提升更敏感，幂律指数约为0.74，增加样本数显著改善模型表现。计算资源限制下，验证损失也遵循幂律关系，最大模型在约1.4×10^7样本训练后达到预测性能。数据重复率实验证明，有限数据环境下，重复训练对性能影响有限，模型在较少新数据下仍能保持较好性能。这些结果验证了规模定律的普适性，为未来大规模驾驶视频模型提供了科学依据。
Applications
该模型可用于自动驾驶中的虚拟场景生成、路径规划和安全测试，尤其在真实数据有限的情况下，通过长时间训练实现更高质量的视频合成。未来，结合多模态信息，可提升模型的泛化能力和场景复杂度，推动自动驾驶系统的虚拟仿真和安全验证。模型还可应用于交通模拟、虚拟训练和智能交通管理，为行业提供高效、可靠的技术支持。
Limitations & Outlook
模型训练成本高昂，硬件资源需求巨大，实际部署存在推理速度瓶颈。数据集虽大但仍有限，未来需扩展多源、多模态数据以提升泛化能力。模型在极端复杂环境中的表现尚未充分验证，存在泛化不足的风险。未来需研究模型压缩和加速技术，降低硬件门槛，增强实际应用的可行性。
Plain Language Accessible to non-experts
想象你在一家厨房做饭，食材代表数据，厨具代表模型，烹饪时间代表训练时间。你可以用少量食材做出简单菜肴，也可以用大量食材和更复杂的厨具做出丰富的菜肴。这个研究就像是在探索用不同的厨具和食材组合，做出最好吃的菜肴的规律。模型参数越大，就像用更高级的厨具，能做出更复杂的菜。训练时间越长，就像烹饪时间越久，菜越入味。数据就像食材的多样性，越丰富越好，但如果反复用同样的食材，味道也可以变得更好。研究发现，长时间烹饪比换更好的厨具更重要，尤其在食材有限的情况下。最终，他们用这些规律训练了一个超级厨师（模型），能做出比以前更美味的菜肴，为自动驾驶的虚拟场景提供了强大工具。
ELI14 Explained like you're 14
想象你在厨房里做饭，你有很多食材（数据），用不同的厨具（模型）和烹饪时间（训练时间）可以做出不同的菜。这个研究就像是在找出用多少厨具、多少时间、多少食材，能做出最好吃的菜。科学家们发现，用更复杂的厨具（大模型）可以做出更好吃的菜，但花费的时间（训练时间）越长，效果越明显，比换厨具更重要。即使食材有限，只要多花点时间反复烹饪，菜也能变得更好吃。最后，他们用这些规律，训练了一个超级厨师（大模型），能做出比以前更美味的菜，为自动驾驶的虚拟场景提供了强大工具。是不是很酷？这就像在厨房里找到最聪明的做菜秘诀一样！
Abstract
Video generation for autonomous driving cannot follow the web-scale route: driving data is expensive to collect, bound by privacy requirements, and cannot be scraped at will, so models must make the most of a fixed corpus. We present a systematic scaling-law study of video diffusion models trained from scratch on driving data: a family of models from 1M to 9B parameters, trained at different exposures on up to 5,500 hours of driving. Validation loss follows consistent power laws in both model size and training exposure, answering the questions that shape a training budget: whether compute is better spent on longer training or on a larger model, and whether more data is needed. Loss improves much faster with training exposure than with model size, making longer training the most effective way to improve a fixed model under limited compute. However, larger models continue to achieve lower asymptotic loss, so compute-optimal scaling still favors increasing model size when sufficient compute and data are available. Guided by these laws, we train a 9B-parameter model, to our knowledge the largest video diffusion model trained from scratch on driving data: it sets a new open-source state of the art for driving video generation, as measured on nuScenes. Our code and pretrained models are available at https://github.com/valeoai/VATIX. NATIX is separately releasing the underlying driving data in stages.