ScalingNoise: Scaling Inference-Time Search for Generating Infinite Videos
ScalingNoise improves video generation quality and consistency by guiding noise selection.
Key Findings
Methodology
ScalingNoise is an inference-time search strategy that improves video generation quality by guiding noise selection. It uses a reward model to evaluate the long-term value of noise candidates and samples from a tilted noise distribution to enhance video consistency and visual diversity.
Key Results
- On the VBench dataset, ScalingNoise improved video consistency and quality, with background consistency increasing by 1.22% and image quality by 2.91%.
- On the UCF-101 dataset, ScalingNoise achieved an FVD score of 539.2, outperforming other methods.
- Ablation studies demonstrated the efficiency and accuracy of the one-step denoising evaluation method.
Significance
This study introduces an inference-time noise selection strategy that significantly improves the quality and consistency of long video generation, addressing existing methods' consistency issues and providing new solutions for resource-constrained environments.
Technical Contribution
ScalingNoise offers new theoretical guarantees and engineering possibilities through guided noise selection and one-step denoising evaluation strategy, significantly differing from existing state-of-the-art methods.
Novelty
ScalingNoise is the first to introduce inference-time noise selection strategy in video generation, providing longer-term quality and consistency guarantees compared to existing methods.
Limitations
- In highly dynamic scenes, noise selection may not completely avoid cumulative errors.
- Additional computational resources are required for noise evaluation.
Future Work
Future work could explore more efficient noise evaluation methods and applications in other generation tasks.
AI Executive Summary
Video generation models have significant impacts on film production, game development, and more. However, existing methods face challenges in maintaining consistency and quality in long video generation. ScalingNoise improves video generation consistency and visual quality through guided noise selection and reward model evaluation. Experiments show that this method performs excellently across multiple benchmark datasets, providing new solutions for long video generation. Nonetheless, further optimization of noise evaluation methods is needed to reduce computational resource consumption.
Deep Analysis
Background
Video generation technology has made significant progress in recent years, especially with the application of diffusion models. Existing research mainly focuses on optimization during training, while noise selection during inference has not been fully addressed.
Core Problem
The core problem in long video generation is how to maintain video consistency and quality under limited resources. Noise selection significantly impacts video generation results.
Innovation
ScalingNoise provides a new inference-time search strategy through guided noise selection and reward model evaluation, effectively solving consistency issues in long video generation.
Methodology
- �� Use reward model to evaluate the long-term value of noise candidates
- �� Sample from tilted noise distribution to enhance video consistency
- �� One-step denoising evaluation strategy reduces computational resource consumption
Experiments
Experiments were conducted on VBench and UCF-101 datasets to evaluate video consistency and quality. Multiple baseline methods were compared, and ablation studies were performed to verify the method's effectiveness.
Results
ScalingNoise performed excellently across multiple datasets, significantly improving video consistency and quality. Ablation studies verified the efficiency and accuracy of the one-step denoising evaluation method.
Applications
This method can be applied to long video generation in film production, game development, and more, especially suitable for resource-constrained environments.
Limitations & Outlook
In highly dynamic scenes, noise selection may not completely avoid cumulative errors. Further optimization of noise evaluation methods is needed to reduce computational resource consumption.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen, and noise is like seasoning. Each seasoning has a different flavor, affecting the final dish. ScalingNoise is like a smart chef who knows how to choose the best seasoning to make the dish tastier. With one-step denoising evaluation, the chef can quickly judge the seasoning's effect, saving time and resources.
ELI14 Explained like you're 14
Imagine you're playing a game with many levels, each with different challenges. ScalingNoise is like a super helper, guiding you to choose the best gear to tackle the challenges. It can quickly evaluate the gear's effect, helping you perform better in the game and score higher!
Glossary
Diffusion Model
A generative model that creates data by progressively denoising. Used as the foundational model for video generation.
Used as the foundational model for video generation.
Noise Selection
Choosing appropriate noise during generation to improve result quality.
Core mechanism of ScalingNoise.
Reward Model
Evaluates the long-term value of noise candidates.
Key component guiding noise selection.
One-step Denoising
A method for quickly evaluating noise, reducing computational resource consumption.
Strategy to improve evaluation efficiency.
Tilted Noise Distribution
A weighted noise distribution used to enhance candidate noise quality.
Used for sampling high-quality noise candidates.
Open Questions Unanswered questions from this research
- 1 How to further improve noise selection accuracy in highly dynamic scenes?
- 2 Can this method be applied to other generation tasks, such as image generation?
Applications
Immediate Applications
Film Production
Improves video generation quality, assisting filmmakers in creating more consistent long videos.
Long-term Vision
Game Development
Applies this method in game development to generate high-quality game animations and scenes.
Abstract
Video diffusion models (VDMs) facilitate the generation of high-quality videos, with current research predominantly concentrated on scaling efforts during training through improvements in data quality, computational resources, and model complexity. However, inference-time scaling has received less attention, with most approaches restricting models to a single generation attempt. Recent studies have uncovered the existence of "golden noises" that can enhance video quality during generation. Building on this, we find that guiding the scaling inference-time search of VDMs to identify better noise candidates not only evaluates the quality of the frames generated in the current step but also preserves the high-level object features by referencing the anchor frame from previous multi-chunks, thereby delivering long-term value. Our analysis reveals that diffusion models inherently possess flexible adjustments of computation by varying denoising steps, and even a one-step denoising approach, when guided by a reward signal, yields significant long-term benefits. Based on the observation, we proposeScalingNoise, a plug-and-play inference-time search strategy that identifies golden initial noises for the diffusion sampling process to improve global content consistency and visual diversity. Specifically, we perform one-step denoising to convert initial noises into a clip and subsequently evaluate its long-term value, leveraging a reward model anchored by previously generated content. Moreover, to preserve diversity, we sample candidates from a tilted noise distribution that up-weights promising noises. In this way, ScalingNoise significantly reduces noise-induced errors, ensuring more coherent and spatiotemporally consistent video generation. Extensive experiments on benchmark datasets demonstrate that the proposed ScalingNoise effectively improves long video generation.