DiReCT: Disentangled Regularization of Contrastive Trajectories for Physics-Refined Video Generation
DiReCT improves VideoPhy physical commonsense score by 16.7% without increasing training time.
Key Findings
Methodology
DiReCT disentangles semantic and physical behavior through macro and micro contrastive signals. Macro contrastive signals draw negatives from semantically distant regions, while micro signals construct negatives with shared semantics but differing physical behavior. A velocity-space distributional regularizer prevents catastrophic forgetting of visual quality.
Key Results
- DiReCT improves physical commonsense score on VideoPhy by 16.7%, outperforming baseline and SFT by 11.3%.
- Experiments demonstrate DiReCT enhances physics consistency without increasing training time.
- The velocity-space distributional regularizer prevents catastrophic forgetting of pretrained visual quality.
Significance
This study addresses physics consistency in video generation by disentangling semantic and physical behavior, enhancing applicability in academia and industry. It offers a new solution for improving the physical plausibility of generated videos.
Technical Contribution
DiReCT provides new theoretical guarantees by disentangling contrastive signals, avoiding semantic-physics entanglement. It introduces new engineering possibilities for enhancing physics consistency in video generation.
Novelty
DiReCT is the first to disentangle semantic and physical behavior using macro and micro contrastive signals, addressing semantic-physics entanglement. It offers a more refined contrastive learning mechanism compared to existing methods.
Limitations
- DiReCT may face challenges in complex scenarios, especially those involving multiple physical behaviors.
- The velocity-space distributional regularizer may require further optimization for different datasets.
Future Work
Future research can explore DiReCT's application in more complex scenarios and optimize the velocity-space distributional regularizer for different datasets.
AI Executive Summary
Video generation technologies often violate basic physics when producing high-fidelity outputs. Existing methods fail to distinguish between physically consistent and impossible dynamics. DiReCT proposes a lightweight post-training framework by disentangling semantic and physical behavior. The method improves physics consistency through macro and micro contrastive signals. Experimental results show DiReCT enhances physical commonsense scores on the VideoPhy dataset while preventing visual quality forgetting. This study provides a new solution for video generation with broad application prospects.
Deep Analysis
Background
Video generation technologies have made significant progress in recent years, but generated videos often violate basic physics. Existing methods fail to effectively distinguish between physically consistent and impossible dynamics, leading to insufficient physical plausibility in generated videos. Contrastive learning offers a solution, but semantic-physics entanglement poses challenges in text-conditioned settings.
Core Problem
Video generation technologies often violate basic physics when producing high-fidelity outputs. Existing methods fail to distinguish between physically consistent and impossible dynamics, leading to insufficient physical plausibility. Semantic-physics entanglement makes contrastive learning difficult to apply effectively in text-conditioned settings.
Innovation
DiReCT disentangles semantic and physical behavior through a lightweight post-training framework. Macro contrastive signals draw negatives from semantically distant regions, while micro signals construct negatives with shared semantics but differing physical behavior. A velocity-space distributional regularizer prevents catastrophic forgetting of visual quality.
Methodology
- �� Use macro contrastive signals to draw negatives from semantically distant regions
- �� Use micro contrastive signals to construct negatives with shared semantics but differing physical behavior
- �� Velocity-space distributional regularizer prevents catastrophic forgetting of visual quality
Experiments
Experiments use the VideoPhy dataset, comparing DiReCT with baseline and SFT methods on physical commonsense scores. The velocity-space distributional regularizer prevents catastrophic forgetting of pretrained visual quality.
Results
DiReCT improves physical commonsense score on VideoPhy by 16.7%, outperforming baseline and SFT by 11.3%. Experiments demonstrate DiReCT enhances physics consistency without increasing training time.
Applications
DiReCT can be applied to improve physics consistency in video generation, suitable for scenarios requiring high physical plausibility, such as education and entertainment industries.
Limitations & Outlook
DiReCT may face challenges in complex scenarios, especially those involving multiple physical behaviors. The velocity-space distributional regularizer may require further optimization for different datasets.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. Each step needs to follow certain physical rules, like the boiling point of water and cooking time for ingredients. Video generation is like cooking, but sometimes it forgets these basic rules, creating unrealistic scenes. DiReCT is like a smart assistant that helps video generation remember these physical rules. By disentangling semantic and physical behavior, DiReCT ensures each step is reasonable, just like ensuring each dish is delicious.
ELI14 Explained like you're 14
Imagine you're playing a game where characters can fly, but sometimes they fly too high, breaking the game's physics rules. DiReCT is like the game's manager, ensuring characters follow physics rules when flying. By disentangling semantic and physical behavior, DiReCT ensures every action in the game is reasonable, just like ensuring characters don't fly off the screen.
Glossary
Flow-matching
A video generation technique ensuring temporal consistency of outputs.
Used to generate temporally consistent video outputs.
Contrastive learning
A machine learning method that improves model performance by distinguishing samples under different conditions.
Used to improve physics consistency in video generation.
Semantic-physics entanglement
The entanglement of semantic and physical behavior, making contrastive learning difficult to apply.
Affects contrastive learning effectiveness in text-conditioned settings.
Velocity-space distributional regularizer
A regularizer preventing catastrophic forgetting of visual quality.
Used to maintain pretrained visual quality.
VideoPhy
A dataset for evaluating physical commonsense in video generation.
Used to evaluate DiReCT's physics consistency.
Open Questions Unanswered questions from this research
- 1 How can DiReCT be applied in more complex scenarios?
- 2 How can the velocity-space distributional regularizer be optimized for different datasets?
Applications
Immediate Applications
Educational Video Generation
DiReCT can be used to generate physically plausible educational videos, helping students understand complex concepts.
Long-term Vision
Entertainment Industry
DiReCT can improve video generation quality in the entertainment industry, enhancing audience experience.
Abstract
Flow-matching video generators produce temporally coherent, high-fidelity outputs yet routinely violate elementary physics because their reconstruction objectives penalize per-frame deviations without distinguishing physically consistent dynamics from impossible ones. Contrastive flow matching offers a principled remedy by pushing apart velocity-field trajectories of differing conditions, but we identify a fundamental obstacle in the text-conditioned video setting: semantic-physics entanglement. Because natural-language prompts couple scene content with physical behavior, naive negative sampling draws conditions whose velocity fields largely overlap with the positive sample's, causing the contrastive gradient to directly oppose the flow-matching objective. We formalize this gradient conflict, deriving a precise alignment condition that reveals when contrastive learning helps versus harms training. Guided by this analysis, we introduce DiReCT (Disentangled Regularization of Contrastive Trajectories), a lightweight post-training framework that decomposes the contrastive signal into two complementary scales: a macro-contrastive term that draws partition-exclusive negatives from semantically distant regions for interference-free global trajectory separation, and a micro-contrastive term that constructs hard negatives sharing full scene semantics with the positive sample but differing along a single, LLM-perturbed axis of physical behavior; spanning kinematics, forces, materials, interactions, and magnitudes. A velocity-space distributional regularizer helps to prevent catastrophic forgetting of pretrained visual quality. When applied to Wan 2.1-1.3B, our method improves the physical commonsense score on VideoPhy by 16.7% and 11.3% compared to the baseline and SFT, respectively, without increasing training time.