PhysEditWorld: A Large-Scale Dataset Toward Physics-Editable World Models

Research in game world modeling has progressed from early neural simulators like GameGAN and Playable Video Generation to sophisticated systems such as Genie, DIAMOND, and GameNGen, supporting action-conditioned generation, exploration, and long-horizon consistency. However, these models typically learn physics implicitly, without explicit control over physical parameters. Existing datasets like MineRL, Sekai, and YUME focus on exploration and interaction but lack matched physical interventions. Physics benchmarks (PHYRE, CLEVRER) evaluate reasoning and plausibility but do not support direct manipulation of physical variables. Simulation platforms like UE and CARLA offer high-fidelity environments but are limited in scale and physical editing capabilities. The gap remains in datasets that enable explicit, controllable physical parameter variation within interactive scenarios, crucial for advancing editable world models.

Current game and simulation models lack the ability to explicitly control physical parameters like gravity within the same scene and interaction context. This impairs the development of editable virtual worlds where designers can tune physics to achieve desired behaviors. Existing datasets do not provide matched replays under varied physical conditions, making it difficult for models to learn sensitivity and respond accurately to physical edits. Consequently, models can generate visually plausible scenes but fail to reflect changes in physical laws, limiting their utility in game design, physics research, and virtual prototyping. Addressing this requires a dataset that supports systematic variation and evaluation of physical parameters in interactive scenarios.

The key innovation is leveraging UE5’s replay system to generate matched replays across different physical configurations, fixing scene, actions, and camera conditions while varying physical parameters like gravity. This approach ensures direct comparability of behaviors under different physics settings. The dataset integrates multimodal signals—RGB, depth, normals, audio, engine states—and explicit gravity labels, enabling models to learn and respond to physical variations explicitly. The pipeline automates scenario setup, action recording, controlled replays, synchronized data export, and semantic annotation, creating a scalable platform for physics-aware modeling. This combination of scene consistency, multimodal richness, and explicit physical control distinguishes PhysEditWorld from prior resources.

• �� Convert artist-designed UE5 levels into replayable scenarios via an in-editor plugin, registering scene assets, controllers, cameras.
• �� Record interaction actions using UE5’s Enhanced Input System, capturing semantic action sequences.
• �� During simulation, fix scene, action, and camera conditions, then vary physical parameters (e.g., gravity) to generate multiple matched replays.
• �� Use UE5’s replay system to ensure scene and interaction consistency, while changing only physical variables.
• �� Synchronize multimodal signals—RGB, depth, normals, audio, engine logs—across all variants.
• �� Export data using Movie Render Queue, aligning frames with engine logs for precise temporal correspondence.
• �� Annotate each replay with semantic captions and physical labels, filtering out failures.
• �� Organize data into matched groups, enabling direct comparison of behaviors under different physics settings.

Models are trained and evaluated on PhysEditWorld’s datasets, with splits ensuring scene and interaction consistency across gravity variants. Fine-tuning is performed with LoRA on attention modules, using 5-second clips at 30 FPS. Evaluation metrics include fall-speed proxies derived from VGGT trajectories, measuring the model’s sensitivity to gravity changes. Experiments compare zero-shot models with fine-tuned versions, demonstrating substantial improvements in gravity alignment and physical response. Additional tests involve first-person world models, verifying whether models can generate gravity-dependent behaviors like free-fall. Ablation studies assess the impact of multimodal signals and explicit labels, confirming the dataset’s effectiveness in enhancing physics awareness.

Fine-tuned models show a jump in gravity alignment from 33% to 100%, with R2 scores on fall acceleration increasing from 0.066 to 0.57, indicating accurate physical response. Behaviorally, models generate realistic free-fall and jump behaviors under varied gravity, matching physical expectations. Quantitative metrics confirm improved sensitivity and response consistency. The dataset enables models to learn physics as an explicit, controllable variable, facilitating applications in game design, physics simulation, and virtual environment customization. These results demonstrate PhysEditWorld’s potential to advance physics-aware generative modeling significantly.

The dataset supports development of physics-sensitive generative models for game engines, VR environments, and robotics simulations. Developers can use it to create more realistic, physically controllable virtual worlds, enabling dynamic scene editing and physical parameter tuning. It also provides a benchmark for evaluating physics-aware generative and world models, fostering innovation in interactive AI, virtual prototyping, and scientific visualization. Long-term, this work could lead to fully editable virtual environments where physical laws are as malleable as visual assets, revolutionizing virtual reality experiences and automated scene generation.

Currently limited to gravity as the only explicit physical parameter; future work should incorporate friction, air resistance, and other physical attributes. The reliance on UE5 constrains scene complexity and scalability, impacting large-scale data collection. Model understanding of complex physical interactions remains imperfect; integrating explicit physics engines could improve fidelity. Additionally, the dataset focuses on controlled scenarios, and real-world complexity may pose challenges for generalization. Addressing these limitations will be crucial for broader applicability and robustness.