PhyPrompt: RL-based Prompt Refinement for Physically Plausible Text-to-Video Generation
Proposes PhyPrompt, an RL-based two-stage prompt refinement framework, achieving 40.8% joint success with 7B parameters.
Key Findings
Methodology
This study employs a two-stage training approach: first, supervised fine-tuning of Qwen2.5 on a physics Chain-of-Thought (CoT) dataset to enhance physical reasoning; second, applying Group Relative Policy Optimization (GRPO) with a dynamic reward curriculum to optimize prompt generation. The reward schedule gradually shifts focus from semantic fidelity to physical commonsense, enabling a balanced, progressive learning process. The method keeps the video generator fixed, training only a lightweight prompt rewriter, ensuring broad compatibility across different T2V architectures. This approach effectively encodes domain-specific physical reasoning into prompts, facilitating zero-shot transfer and consistent improvements.
Key Results
- On VideoPhy2, PhyPrompt-7B achieves 40.8% joint success, an 8.6 percentage point increase over baseline prompts, with physical commonsense scores rising by 11 points (55.8% to 66.8%) and semantic adherence improving by 4.4 points (43.4% to 47.8%).
- Across diverse models (Lavie, VideoCrafter2, CogVideoX-5B), the method shows up to 16.8% improvement in joint metrics, confirming its model-agnostic transferability.
- Ablation studies demonstrate that the dynamic reward schedule outperforms static weighting, and the two-stage training yields significant synergistic gains, outperforming single-objective approaches.
Significance
This work addresses the critical gap in physical plausibility in T2V models, demonstrating that domain-specific reinforcement learning with curriculum strategies can substantially improve physical realism without increasing model size. The proposed prompt refinement method enables more reliable, scalable, and physically consistent video synthesis, which is vital for applications in robotics, simulation, and scientific visualization. Its zero-shot transferability across architectures highlights its practicality. The approach paves the way for future research integrating physical reasoning into large-scale generative models, fostering more natural and trustworthy virtual content creation.
Technical Contribution
The key technical innovation lies in integrating physics-aware Chain-of-Thought fine-tuning with a novel GRPO-based prompt optimization guided by a dynamic reward curriculum. This combination allows the model to learn a balanced representation of semantic and physical constraints, avoiding the pitfalls of static multi-objective optimization. The lightweight prompt rewriter, trained solely on RL feedback, achieves broad compatibility without retraining the generator. The method demonstrates that task-specific, curriculum-driven reinforcement learning can outperform large-scale pretraining, providing a parameter-efficient pathway to domain-aware content generation.
Novelty
This is the first work to combine physics-focused Chain-of-Thought training with a curriculum-based RL prompt refinement specifically for text-to-video generation. Unlike prior methods that rely on internal model modifications or static multi-objective balancing, this approach dynamically adjusts reward emphasis, enabling the discovery of novel prompt structures that enhance physical plausibility. Its generator-agnostic design and zero-shot transfer capability distinguish it from existing prompt tuning and enhancement techniques, marking a significant step forward in physics-aware content synthesis.
Limitations
- The method depends on the coverage of the physics Chain-of-Thought dataset, which may limit generalization to untrained physical scenarios.
- Complex dynamic scenes with multiple interacting objects still pose challenges, requiring further integration with physical simulation engines.
- Sensitivity to reward scheduling parameters may affect stability and performance, necessitating careful tuning for different tasks.
Future Work
Future directions include developing adaptive reward mechanisms that automatically tune the curriculum based on model feedback, integrating physical simulation for more complex interactions, and extending the approach to other modalities such as 3D content. Additionally, exploring multi-task training to incorporate multiple physical laws simultaneously could further enhance realism. Combining this with real-world physical data and physics engines may lead to more robust, high-fidelity virtual environments suitable for scientific research and industrial applications.
AI Executive Summary
Recent advances in text-to-video (T2V) generation have achieved remarkable visual quality, yet the physical plausibility of generated content remains a significant challenge. Many models produce videos where objects violate fundamental physical laws—objects teleport, pass through each other, or defy gravity—limiting their practical deployment in scientific, robotic, and simulation domains. Addressing this, the present work introduces PhyPrompt, a novel reinforcement learning-based prompt refinement framework designed to automatically enhance the physical realism of generated videos.
The core idea is to leverage a two-stage training process. First, a large language model (Qwen2.5) is fine-tuned on a curated physics Chain-of-Thought dataset, enabling it to reason about physical phenomena such as object motion, force interactions, and causal dynamics. This step imbues the model with domain-specific reasoning capabilities. Second, the prompt generation policy is refined using Group Relative Policy Optimization (GRPO), an RL algorithm that samples multiple candidate prompts per input and optimizes them based on a composite reward signal.
A key innovation is the dynamic reward curriculum, which gradually shifts focus from semantic fidelity—ensuring the prompt matches user intent—to physical commonsense—ensuring the generated video adheres to physical laws. This staged approach allows the model to first learn to preserve content meaning, then to incorporate physical constraints, leading to a synergistic improvement.
Experimental results demonstrate that PhyPrompt-7B achieves a 40.8% joint success rate on the VideoPhy2 benchmark, outperforming baseline prompts by 8.6 percentage points. It also exhibits strong zero-shot transferability across diverse architectures like Lavie, VideoCrafter2, and CogVideoX-5B, with improvements up to 16.8%. Ablation studies confirm the effectiveness of the curriculum strategy and the two-stage training.
Overall, this work significantly advances the state of physics-aware content generation, providing a scalable, model-agnostic solution that bridges the gap between visual realism and physical plausibility. It opens new avenues for research in physically grounded generative models and broadens the applicability of T2V systems in real-world scenarios.
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房做菜，很多时候你会根据食材的特性和烹饪规律调整步骤，比如知道火不能太大，否则会糊掉。现在，生成视频就像做菜，模型需要理解物理规律，比如重力、碰撞和流动。传统的模型就像只看食谱，做出来的菜可能看起来不错，但味道不对。本文的方法就像请厨师学习这些烹饪技巧，然后用强化训练让厨师自己优化菜谱，确保每次做出来的菜既好看又好吃。通过这种方式，生成的视频不仅漂亮，还符合自然规律，就像真正的厨艺一样自然。
ELI14 Explained like you're 14
想象你在玩一个游戏，你可以用文字告诉游戏角色做什么，比如“让球从高处滚下来”。但如果你没有告诉它重力的规则，它可能会让球飞到天上或者穿墙。这个研究就像教游戏角色理解重力和碰撞的规则，然后让它自己学习怎么用正确的方式做事。科学家用一种叫强化学习的方法，就像给角色设定奖励，让它学会做符合自然规律的动作。这样，生成的视频就像真实世界一样，物体不会突然飞走或穿墙。研究还发现，用这种方法训练的角色，不仅在一个游戏里表现好，还能在其他不同的游戏中也表现得很好。未来，这项技术可以让虚拟世界变得更真实，像动画电影一样自然，也可以帮助机器人更聪明地理解世界。
Abstract
State-of-the-art text-to-video (T2V) generators frequently violate physical laws despite high visual quality. We show this stems from insufficient physical constraints in prompts rather than model limitations: manually adding physics details reliably produces physically plausible videos, but requires expertise and does not scale. We present PhyPrompt, a two-stage reinforcement learning framework that automatically refines prompts for physically realistic generation. First, we fine-tune a large language model on a physics-focused Chain-of-Thought dataset to integrate principles like object motion and force interactions while preserving user intent. Second, we apply Group Relative Policy Optimization with a dynamic reward curriculum that initially prioritizes semantic fidelity, then progressively shifts toward physical commonsense. This curriculum achieves synergistic optimization: PhyPrompt-7B reaches 40.8\% joint success on VideoPhy2 (8.6pp gain), improving physical commonsense by 11pp (55.8\% to 66.8\%) while simultaneously increasing semantic adherence by 4.4pp (43.4\% to 47.8\%). Remarkably, our curriculum exceeds single-objective training on both metrics, demonstrating compositional prompt discovery beyond conventional multi-objective trade-offs. PhyPrompt outperforms GPT-4o (+3.8\% joint) and DeepSeek-V3 (+2.2\%, 100$\times$ larger) using only 7B parameters. The approach transfers zero-shot across diverse T2V architectures (Lavie, VideoCrafter2, CogVideoX-5B) with up to 16.8\% improvement, establishing that domain-specialized reinforcement learning with compositional curricula surpasses general-purpose scaling for physics-aware generation.