Distilling Physical Priors into Streaming World Models
PhyS framework improves streaming world models' physical consistency by distilling physical priors, enhancing benchmarks like PhysicsIQ.
Key Findings
Methodology
PhyS framework consists of three stages: physics-aware supervised fine-tuning using the PhyS-120K dataset to acquire physical priors; distilling these priors into a lightweight causal generator; optimizing the generator's physical consistency through online reinforcement learning and Temporal Credit Routing.
Key Results
- Result 1: PhyS framework improved Wan2.1-14B teacher model's performance by 18.2% on PhysicsIQ.
- Result 2: On VideoPhy benchmark, physical commonsense and semantic adherence improved by 16.0% and 1.5%, respectively.
- Result 3: PhyGenBench average score increased by 18.5%.
Significance
This research addresses long-standing issues of physical consistency in streaming world models, significantly impacting academia and industry in physics-aware video generation.
Technical Contribution
PhyS framework introduces a novel optimization strategy through Temporal Credit Routing, significantly enhancing causal video generators' physical consistency, opening new engineering possibilities.
Novelty
PhyS is the first to distill physical priors into streaming world models and optimize physical consistency through Temporal Credit Routing, offering significant innovation compared to existing methods.
Limitations
- Limitation 1: The method shows limited performance in handling complex optics and material interactions.
- Limitation 2: Physical priors may be lost during causal distillation.
Future Work
Future research could explore more complex physical phenomena and material interactions, further optimizing causal generators' physical consistency.
AI Executive Summary
Streaming world models often violate basic physical constraints when predicting future visual states. The PhyS framework addresses this issue through a three-stage process. First, it uses the PhyS-120K dataset for physics-aware supervised fine-tuning to acquire physical priors. Second, these priors are distilled into a lightweight causal generator. Finally, the generator's physical consistency is optimized through online reinforcement learning and Temporal Credit Routing. Experimental results show significant improvements on benchmarks like PhysicsIQ, VideoPhy, and PhyGenBench, highlighting its importance in physics-aware video generation. Despite these advancements, the method has limitations in handling complex optics and material interactions, which future research could address.
Deep Analysis
Background
Streaming world models have made significant progress recently, but their predictions often violate physical constraints. Existing methods primarily distill bidirectional DiT models to generate causal videos, but this approach has limitations in physical consistency.
Core Problem
Streaming world models often violate physical constraints during long-horizon predictions, leading to poor physical consistency. Solving this problem is crucial for improving the realism and utility of video generation.
Innovation
The PhyS framework innovatively improves streaming world models' physical consistency through a three-stage process: physics-aware supervised fine-tuning, causal distillation, and Temporal Credit Routing.
Methodology
- �� Physics-aware supervised fine-tuning: Fine-tune using the PhyS-120K dataset to acquire physical priors.
- �� Causal distillation: Distill physical priors into a lightweight causal generator.
- �� Temporal Credit Routing: Optimize the generator's physical consistency through online reinforcement learning.
Experiments
Experiments use the PhyS-120K dataset for fine-tuning and evaluate on benchmarks like PhysicsIQ, VideoPhy, and PhyGenBench, demonstrating significant improvements with the PhyS framework.
Results
The PhyS framework improved performance by 18.2% on PhysicsIQ and showed significant enhancements on VideoPhy and PhyGenBench, highlighting its advantages in physical consistency.
Applications
The PhyS framework can be used for physics-aware video generation, suitable for scenarios requiring high physical consistency, such as simulation and education.
Limitations & Outlook
Despite advancements in physical consistency, PhyS has limitations in handling complex optics and material interactions, which future research could further optimize.
Plain Language Accessible to non-experts
Imagine a kitchen where a chef is preparing a complex dinner. Each step must follow specific rules, like chopping vegetables, boiling soup, and baking bread. Streaming world models are like this chef, and the PhyS framework helps the chef follow physical rules to ensure each dish meets expectations. Through physics-aware supervised fine-tuning, the chef learns how to chop and cook correctly. Causal distillation is like the chef passing these skills to an assistant, ensuring each step is executed correctly. Finally, Temporal Credit Routing helps the chef maintain consistency between dishes, ensuring the entire dinner is perfectly presented.
ELI14 Explained like you're 14
Hey, friends! Imagine you're playing a super cool game with lots of physics effects, like balls rolling and liquids flowing. Streaming world models are like the game's characters, needing to follow physics rules to make the game look real. The PhyS framework is like a super tool in the game, helping characters follow physics rules. Through physics-aware supervised fine-tuning, characters learn how to move correctly. Causal distillation is like characters passing these skills to teammates, ensuring the whole game follows physics rules. Finally, Temporal Credit Routing helps characters stay consistent between levels, making the game more fun!
Glossary
Streaming World Models
Models that predict future visual states, requiring long-term physical consistency.
Used for generating physically consistent video sequences.
Physics-aware Supervised Fine-tuning
Fine-tuning models using a dataset of physical interaction videos to acquire physical priors.
Used to enhance model's physical consistency.
Causal Distillation
Distilling physical priors from bidirectional models into lightweight causal generators.
Used for generating causal video sequences.
Temporal Credit Routing
An optimization strategy that routes physical feedback signals through temporal windows.
Used to enhance generator's physical consistency.
PhyS-120K Dataset
A real-world dataset containing 120,804 physical interaction videos.
Used for physics-aware supervised fine-tuning.
Open Questions Unanswered questions from this research
- 1 How to maintain physical consistency in complex optics and material interactions? Current methods show limited performance in these scenarios.
Applications
Immediate Applications
Educational Simulation
The PhyS framework can be used for educational simulations, helping students understand physical phenomena.
Long-term Vision
Virtual Reality
Apply the PhyS framework in virtual reality to enhance the realism of user experiences.
Abstract
Streaming world models predict future visual states online while maintaining physically coherent dynamics over long horizons. However, their rollouts often violate basic physical constraints. A common approach distills pretrained bidirectional DiTs into few-step causal generators. However, this paradigm suffers from two fundamental limitations: generic bidirectional teachers acquire limited physical priors from visually oriented pretraining, and the limited priors suffer further loss during bidirectional-to-causal distillation. We present PhyS, a three-stage framework for distilling physical priors into streaming world models. To acquire physical priors from real-world interactions, we construct PhyS-120K, a dataset of 120K real-world physical-interaction videos spanning rigid-body dynamics, soft-body deformation, fluid phenomena, and phase transitions. Each video is annotated with structured descriptions of object properties and causal state transitions. Physics-aware supervised fine-tuning injects the physical priors into a bidirectional 14B DiT teacher, which we then distill into a lightweight 1.3B causal DiT for few-step autoregressive streaming generation. Finally, we use online reinforcement learning to incentivize the distilled model to generate physically plausible rollouts and further propose Temporal Credit Routing (TCR) to address temporal credit assignment. TCR evaluates physical consistency over overlapping temporal windows and routes the resulting group-relative advantages to temporally aligned denoising actions. On PhysicsIQ, PhyS improves the Wan2.1-14B teacher by 18.2\% and the Self Forcing, Rolling Forcing, and Causal Forcing by 23.7\%, 14.8\%, and 31.4\%, respectively. Results also improve the physics-aware video benchmarks VideoPhy, VideoPhy2, and PhyGenBench. The dataset, code, and more sample videos are available on our Project Page.