PhysGM: Large Physical Gaussian Model for Feed-Forward 4D Synthesis

Recent years have seen significant progress in static scene representation, notably with 3D Gaussian Splatting (3DGS), which enables efficient multi-view synthesis. However, extending these static models to dynamic, physically plausible 4D content remains challenging. Traditional approaches rely on dense multi-view reconstructions followed by scene-specific optimization, which is computationally intensive and unsuitable for real-time applications. Deep learning methods like NeRF, SDF, and implicit functions have advanced scene generation but lack explicit physical reasoning, often producing physically implausible animations. PhysGaussian and related works attempted to incorporate physics via optimization-heavy processes, but their scalability is limited. The field thus faces a trade-off between realism, speed, and scalability, motivating the development of models that can predict dynamic scenes directly from minimal input.

The core challenge is achieving fast, physically consistent 4D scene synthesis from a single image without scene-specific optimization. Existing methods are either too slow, requiring per-scene tuning, or lack physical realism. The bottleneck lies in the iterative reconstruction and optimization procedures, which hinder scalability and real-time deployment. Moreover, naive concatenation of geometry and physics modules ignores embedded physical cues in appearance, leading to suboptimal results. Addressing these issues necessitates a unified, end-to-end model capable of instant prediction of scene geometry, physical properties, and their dynamic evolution, while maintaining high visual and physical fidelity.

This work introduces several key innovations: 1) a transformer-based joint prediction architecture that estimates scene geometry and physical parameters simultaneously, 2) a two-stage training paradigm combining large-scale supervised learning with preference-based fine-tuning (DPO), 3) the creation of PhysAssets, a large annotated dataset supporting physical and geometric supervision, 4) an optimization-free pipeline that directly drives physics simulation from predicted parameters, and 5) a novel use of probabilistic physical property modeling to capture uncertainty and enable diverse scene generation. These innovations collectively enable real-time, high-fidelity, physically grounded 4D scene synthesis from minimal input.

• �� Input: single RGB image and camera parameters. • Feature extraction: use DINOv3 encoder to obtain image features; encode camera view with Plücker coordinates. • Tokenization: concatenate image features and camera tokens; add learnable global tokens. • Transformer backbone: process tokens through 24-layer transformer to learn scene context. • Geometry prediction: a Dense Prediction Transformer (DPT) head outputs per-pixel Gaussian parameters (μ, Σ, color, alpha). • Physical property prediction: classification head determines material class; regression heads predict mean and variance for Young’s modulus (E) and Poisson ratio (ν). • Sampling: draw physical parameters from learned distributions. • Physics simulation: use predicted parameters to initialize Material Point Method (MPM), updating particle positions and deformation. • Gaussian update: derive covariance from deformation gradient via polar decomposition, ensuring geometry matches physical deformation. • Fine-tuning: apply DPO by generating scene variants, comparing with reference videos, and optimizing preference ranking to improve realism and physical consistency.

• �� Dataset: PhysAssets, with 50K+ annotated 3D objects and corresponding simulation videos. • Baselines: SDS-based OmniPhysGS and DreamerPhysics, which rely on scene-specific optimization. • Metrics: PSNR, SSIM, LPIPS for reconstruction; CLIPscore for semantic similarity; user preference rate (UPR) for perceptual realism. • Training: two-stage process with large-scale supervised learning followed by DPO fine-tuning. • Evaluation: diverse scenarios including multiple materials and complex deformations, testing speed, fidelity, and physical plausibility. • Ablation: removal of DPO or dataset components to assess contribution of each element.

• �� Achieved scene generation in under 1 minute from a single image, outperforming SDS methods by over 30x in speed. • Quantitative metrics show superior scores: PSNR increased by 10%, SSIM by 8%, LPIPS decreased by 0.05, CLIPscore improved by 15%. • User studies indicate 20% higher preference for PhysGM outputs, demonstrating enhanced perceptual realism. • Ablation results confirm that the two-stage training and preference-based fine-tuning significantly improve physical and visual quality, with the model generalizing well across diverse scenarios.

• �� Immediate: rapid creation of virtual environments for gaming, AR/VR, and digital content, where users can generate dynamic scenes from a single photo. • Long-term: enabling autonomous robots to simulate and understand physical interactions in real time, advancing virtual prototyping, and automating content creation pipelines, reducing manual effort and costs.

• �� The model's accuracy diminishes with highly complex, non-rigid, or multi-object scenes, where physical parameters are harder to predict precisely. • Dependence on large, annotated datasets limits generalization to unseen materials or environments. • Computational costs, though reduced, still pose challenges for ultra-high-resolution or multi-object scenarios, requiring further optimization for deployment in resource-constrained settings.