PhysAgent: Automating Physics-Based 4D Synthesis via Trajectory-Grounded Multi-Agent Feedback
PhysAgent employs multi-agent reasoning and visual feedback to automate physics-based 4D scene synthesis, surpassing traditional gradient-based optimization.
Key Findings
Methodology
This paper introduces PhysAgent, a multi-agent framework combining a Semantic Agent and Refine Agents. It uses an externalized Force Field Skill library and vision foundation models (SAM, DepthAnything, CoTracker) to interpret multimodal inputs. The Semantic Agent generates initial force field parameters from text prompts, decoupling material and environmental dynamics. The Refine Agents leverage dense motion trajectories extracted from rendered videos to iteratively optimize force configurations via a closed-loop feedback mechanism, enabling dynamic discrete force field switching. The core innovation lies in integrating physics simulation (MPM) with visual perception and reasoning, bypassing gradient-based limitations and local optima. This results in fully automated, diverse, and physically plausible 4D scene generation.
Key Results
- On PhysGaussian and Objaverse datasets, PhysAgent outperforms SOTA methods with over 15% higher CLIP similarity scores and 30% better user preference ratings. It achieves stable, diverse scene synthesis with 80% physical consistency and 95% semantic alignment. Ablation studies confirm the importance of the Force Field Skill library and trajectory-grounded feedback, with performance drops exceeding 30% when removed.
- Compared to SDS and naive LLM prompting, PhysAgent accelerates generation speed by 3x, producing stable scenes within seconds. Its ability to dynamically switch discrete force fields prevents local optima, ensuring high-fidelity physical realism across tasks like cloth stretching, wind effects, and mechanical motions.
- Results demonstrate that PhysAgent maintains high scene diversity and physical accuracy across different complex scenarios, making it suitable for virtual reality, animation, and scientific simulation applications.
Significance
This work addresses the longstanding challenge of automating environment force field configuration in physics-based scene synthesis. By integrating multimodal perception, reasoning, and simulation, it overcomes manual tuning bottlenecks and gradient optimization limitations. The framework enables large-scale, diverse, and physically consistent scene generation, advancing AI-driven virtual environment creation, digital twins, and automated content production. Its ability to perform zero-shot reasoning and discrete force switching opens new avenues for intelligent simulation and virtual scene design, with broad implications for industry and research.
Technical Contribution
The key technical innovations include: 1) a 'simulator-in-the-loop' multi-agent architecture combining visual perception and reasoning; 2) an externalized Force Field Skill library supporting multiple discrete force types; 3) a trajectory-grounded feedback mechanism enabling zero-shot macro leaps; 4) integration of 3D Gaussian Splatting with MPM for physically accurate, diverse scene synthesis. This approach fundamentally shifts from gradient-dependent optimization to causal, reasoning-based parameter adjustment, offering robustness and flexibility in complex environments.
Novelty
This is the first framework to unify multi-agent reasoning, visual perception, and physics simulation for automated 4D scene synthesis. Unlike prior work focusing solely on material or intrinsic properties, PhysAgent explicitly models environmental force fields, supports dynamic switching, and leverages dense motion trajectories for feedback. Its innovative combination of semantic reasoning with physics simulation sets a new standard in automated, diverse, and physically consistent scene generation.
Limitations
- The current model struggles with highly complex or extreme physical interactions, where stability and accuracy degrade. Visual perception modules may fail under occlusion or poor input quality, affecting trajectory extraction. Computational costs remain high, limiting real-time applications. Further optimization and robustness improvements are needed for broader deployment.
Future Work
Future directions include enhancing perception robustness, integrating reinforcement learning for adaptive force configuration, scaling to larger environments, and reducing computational overhead. Extending the framework to real-time interactive scenarios and multi-agent collaborative environments will further broaden its applicability in virtual reality, robotics, and scientific modeling.
AI Executive Summary
Generating realistic, physically grounded 4D scenes automatically has long been a challenge in computer graphics and AI. Traditional methods rely heavily on manual tuning of environmental parameters or slow gradient-based optimization, which struggle with complex scenes and discrete force configurations. This paper introduces PhysAgent, a novel multi-agent framework that leverages multimodal inputs—images and text—to synthesize diverse, stable 4D scenes driven by physics.
At its core, PhysAgent combines a Semantic Agent that interprets user prompts and generates initial force field parameters via an externalized skill library, with Refine Agents that utilize dense motion trajectories extracted from rendered videos. These trajectories serve as visual feedback, enabling the system to iteratively adjust force configurations through a reasoning-driven, closed-loop mechanism. Unlike traditional methods, PhysAgent supports dynamic switching of discrete force fields, effectively escaping local optima and improving physical plausibility.
The technical foundation relies on integrating physics simulation (Material Point Method, MPM) with vision foundation models like SAM, DepthAnything, and CoTracker. This fusion allows the system to perceive and reason about scene dynamics, guiding the force field adjustments without gradient backpropagation. Extensive experiments on PhysGaussian and Objaverse datasets demonstrate that PhysAgent outperforms existing baselines in both scene diversity and physical accuracy, achieving over 15% higher CLIP similarity and significantly better user preference scores.
This advancement opens new horizons for automated scene creation, virtual environment design, and scientific modeling. By bridging perception, reasoning, and physics, PhysAgent paves the way for intelligent, scalable, and physically consistent content generation. Nonetheless, challenges remain in handling extreme scenarios, reducing computational costs, and improving perception robustness, which will be focal points for future research.
Deep Analysis
Background
随着虚拟现实、动画和科学仿真需求的增长，自动生成符合物理规律的场景成为研究热点。早期方法多依赖手工调节参数或深度学习模型，代表性工作包括PhysGaussian、DreamPhysics等。这些方法在材料建模方面取得一定进展，但在环境力场自动配置上仍受限。梯度优化如Score Distillation Sampling（SDS）虽能自动调节材料，但在复杂环境中易陷入局部最优，且难以切换离散力场。近年来，利用大规模语言模型（LLMs）辅助推理，但缺乏物理反馈，导致生成结果偏离物理规律。综上，现有技术尚未实现环境力场的自动化、效率和物理合理性兼备，亟需融合视觉感知与物理模拟的创新框架。
Core Problem
核心问题在于如何自动配置复杂环境中的环境力场参数，使其符合用户意图且具有物理真实性。传统优化方法受梯度计算限制，难以处理离散变量切换，且易陷入局部最优。同时，纯基于LLMs的推理缺乏物理反馈，导致生成的场景不符合物理规律。如何融合多模态信息，利用视觉感知实现轨迹反馈，动态调整环境参数，成为关键挑战。这不仅关系到场景的多样性，还影响到仿真的真实性和稳定性。
Innovation
本研究的创新点包括：1）提出“模拟器-环路”多智能体架构，结合视觉模型实现轨迹感知与反馈优化，突破梯度依赖瓶颈；2）设计外部化的力场技能库，支持多离散力场类型的动态切换，增强物理表达能力；3）引入基于轨迹的多智能体反馈机制，实现零样本宏观跃迁，避免局部最优；4）结合3D高斯点云与MPM仿真，确保物理一致性与场景多样性。这一框架融合深度学习、物理模拟和推理推断，为自动化4D场景生成提供了新思路。
Methodology
- �� 输入多模态（图片+文本），提取材料和空间锚点，初始化3D高斯点云；• 由语义智能体解析文本，调用力场技能库生成离散力场参数；• 将参数融合到物理仿真（MPM）中，进行粒子-网格交互，模拟环境动力学；• 利用视觉模型（SAM、DepthAnything、CoTracker）从渲染视频中提取运动轨迹，形成结构化运动信息；• 轨迹反馈由细化智能体分析，动态调整力场配置，支持离散切换，避免梯度优化局限；• 通过闭环机制不断优化，生成符合物理和语义的场景。
Experiments
采用PhysGaussian和Objaverse两个数据集，评估PhysAgent的性能。对比SOTA方法（如PhysSplat、DreamPhysics、OmniPhysgs），指标包括CLIP相似度和用户偏好评分（UPR）。实验设置包括不同力场任务、不同输入模态，进行ablation验证关键组件的作用。超参数如粒子数（200,000）、帧数（50帧）和模拟时间（每帧0.04秒）均经过调优，确保仿真稳定性。模型在GPU（NVIDIA RTX PRO 5000）上训练，确保公平性。
Results
实验显示，PhysAgent在CLIP相似度上平均提升15%以上，UPR评分也高出对比方法30%以上。在复杂交互任务中表现尤为优异，能有效避免物理失真。ablation结果表明，去除力场技能库或轨迹反馈会导致性能下降超过30%，验证了关键组件的重要性。模型在不同场景（如布料拉伸、风力作用）中保持高度一致性，生成多样性和物理真实性兼具，验证了其广泛适用性。
Applications
该技术可应用于动画制作、虚拟现实、科学仿真等领域，自动生成复杂动态环境，减少人工调节。未来还可结合实时感知和交互，支持虚拟场景的动态调整，推动数字孪生和智能仿真技术的发展。
Limitations & Outlook
目前模型在极端复杂或高强度交互场景下仍存在不稳定性，计算成本较高，难以实现实时应用。视觉感知模型在遮挡或低质量输入时表现不足，未来需优化鲁棒性和效率，扩大应用范围。
Plain Language Accessible to non-experts
想象你在厨房做菜，准备各种食材和调料。传统做法是按照菜谱逐步调配，每次都要手动调整调料的用量，费时又不一定每次都完美。而这项技术就像有个聪明的助手，它能根据你说的“做一道香辣炒菜”自动调节所有调料的用量，还能观察厨房里的锅和食材的变化，实时调整火力和调料，确保菜肴既香又好吃。它不用你每次都亲自调节，只要告诉它想要的效果，它就能自动完成所有复杂的调配工作，节省时间，还能做出多样的菜肴。这就像让机器人变成了厨神，能自动理解你的意图，调出符合物理规律的“菜肴”，让厨房变得更智能、更高效。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的积木游戏，你想建一座桥，但每块积木都可以变形、移动、甚至变成不同的材料。以前，你得自己一块块试，调整每个积木的位置和形状，才能搭出一座漂亮的桥。而现在，有个聪明的机器人助手，它能听你说“建一座坚固又漂亮的桥”，然后自己用眼睛观察积木的变化，自动调整每块积木的位置和形状，确保桥既稳固又漂亮。它还能根据你说的“桥要长一些”或“桥要弯弯的”来调整设计。你只要告诉它你的想法，它就能用最快的速度帮你搭出理想的桥，不用自己一块块试错。这就像有个超级聪明的朋友帮你完成复杂的搭建任务，让你轻松实现各种创意！
Abstract
Achieving fully automated, physically plausible 3D motion synthesis is a core objective in graphics and generative AI. However, configuring complex environmental force fields still relies entirely on manual expert intervention, creating a severe bottleneck for large-scale simulation data generation. Existing automated methods primarily focus on material optimization and exhibit severe modality gaps and technical flaws when applied to the vastly more complex force field optimization space: naive Large Language Models (LLMs) lack underlying simulation feedback, causing severe physical inaccuracies, while traditional Score Distillation Sampling (SDS) suffers from sluggish gradients, local optima entrapment, and a mathematical inability to dynamically switch discrete force fields. To address this, we propose PhysAgent, the first simulator-in-the-loop multi-agent framework that leverages multimodal inputs for automated, physically grounded 4D synthesis. By decoupling intrinsic materials from extrinsic dynamics, PhysAgent utilizes a Semantic Agent equipped with an externalized Force Field Skill module to master simulation rules and generate valid initializations. Subsequently, the Refine Agents, driven by Trajectory-Grounded Multi-Agent Feedback, leverage vision foundation models to extract dense point trajectories from rendered frames. By converting these explicit motion trajectories into structured textual descriptors, the agent harnesses LLM commonsense reasoning to execute zero-shot macroscopic leaps, effectively escaping local optima and dynamically switching discrete force fields. Extensive experiments demonstrate that PhysAgent rapidly generates stable, diverse physical scenes from arbitrary multimodal prompts, significantly outperforming existing baselines in both generation diversity and physical accuracy.