BridgeV2W: Bridging Video Generation Models to Embodied World Models via Embodiment Masks

The evolution of robot perception and control has transitioned from rule-based systems to deep learning models. Early methods relied on handcrafted features, which struggled in complex environments. The advent of large-scale video diffusion models like VideoDiffusion and Stable Diffusion provided rich visual priors, enabling more realistic scene synthesis. Recent efforts integrate these models into robotic tasks, aiming to improve scene understanding, motion prediction, and control. However, these approaches often depend on task-specific data, lack multi-view robustness, and struggle with architecture unification across different robot platforms. Addressing these limitations is crucial for scalable embodied AI.

The main challenge lies in bridging the gap between coordinate-space actions, which are low-dimensional and robot-specific, and pixel-space videos generated by pretrained models. This mismatch hampers effective conditioning, especially under viewpoint changes. Additionally, existing methods lack a unified architecture for different robot embodiments, limiting scalability. Static background overfitting further reduces dynamic task learning. Overcoming these issues requires a representation that aligns actions with visual content across views and robots, ensuring generalization and robustness.

BridgeV2W introduces pixel-aligned embodiment masks derived from URDF and camera parameters, serving as spatial conditions for pretrained video models. This approach effectively closes the action-video gap, enhances viewpoint robustness, and unifies architectures across robot types. It adopts a ControlNet-style pathway for embedding masks, preserving pretrained priors while enabling action conditioning. The flow-based loss emphasizes learning dynamic, task-relevant regions, reducing static background interference. This integrated framework supports training on uncalibrated, actionless videos, broadening applicability.

• �� Generate embodiment masks by projecting URDF-based robot models onto camera images using intrinsic/extrinsic parameters. • Encode initial frames and masks with a pretrained VAE, extracting latent features. • Inject masks into a ControlNet pathway connected to the DiT backbone, guiding video generation conditioned on embodiment masks. • Incorporate optical flow estimation (RAFT) to compute motion discrepancies between predicted and ground-truth videos. • Design a combined loss function: diffusion loss for frame fidelity, dynamics consistency for temporal coherence, and flow loss for dynamic regions. • Train on datasets with diverse viewpoints and scenes, validating generalization and robustness across multiple embodiments.

Using DROID and AgiBot-G1 datasets, with 19k and 15k trajectories respectively, the model is trained to generate 25-frame videos conditioned on embodiment masks. Metrics include PSNR, SSIM, LPIPS, FVD, and Mask-IoU, evaluated under in-domain, unseen viewpoints, and unseen scenes. Baselines include IRASim, Cosmos, and EVAC. Ablation studies remove components like pretraining, masks, ControlNet, or flow loss to assess their impact. Results show superior performance in video quality, temporal coherence, and action alignment, confirming the effectiveness of the proposed design.

BridgeV2W achieves PSNR of 22.89 and FVD of 145.2 on DROID, outperforming baselines. Under unseen viewpoints, FVD reduces to 191.3, Mask-IoU increases to 55.3. On AgiBot-G1, PSNR reaches 24.49, FVD drops to 129.5, Mask-IoU is 58.3. Ablation results confirm each component’s critical role, with removal leading to performance drops. The model maintains high-quality, view-robust video generation across diverse scenarios, demonstrating strong generalization.

This framework enables realistic robotic scene simulation, policy evaluation, and goal-conditioned planning without requiring precise geometric calibration. It supports training on large-scale uncalibrated videos, facilitating scalable robot learning. Future integration with reinforcement learning could allow robots to autonomously improve control policies based on generated videos, accelerating deployment in real-world tasks such as industrial automation, service robots, and autonomous vehicles.

Dependence on accurate URDF and camera calibration may limit real-world robustness. Optical flow estimation can falter in occlusion or high-speed scenarios. Computational costs are high due to large model sizes and training complexity. Future work should focus on reducing costs, improving robustness to sensor errors, and extending to more complex, dynamic environments.