World2Act: Latent Action Post-Training from World Model Dynamics

The evolution of robotic control has increasingly relied on world models (WMs) to simulate environment dynamics, with notable works including VAE-based models, flow-based models, and V-JEPA. Early efforts focused on pixel-level predictions and inverse dynamics models (IDMs), but these suffered from visual noise and limited transferability. The advent of latent space modeling, exemplified by UWM and Cosmos Policy, improved robustness but often involved high-dimensional embeddings that caused training instability. Recent trends aim to leverage structured video VAEs like Cosmos-Predict2, which encode visual sequences into low-dimensional latent trajectories, facilitating more stable dynamic modeling. Despite these advances, existing methods still depend on pixel supervision or pseudo-labels, which are sensitive to visual artifacts and can degrade performance, especially in real-world scenarios. This paper builds on these foundations, proposing a latent-space alignment approach that directly transfers WM dynamics into robot action representations, bypassing pixel-level supervision.

Current WM-based post-training methods predominantly depend on pixel-space supervision, which is highly susceptible to visual artifacts, noise, and inconsistencies in generated rollouts. Such issues lead to noisy pseudo-labels and reward signals, impairing policy stability and transferability. This bottleneck hampers the deployment of embodied agents in real-world environments, where visual imperfections are inevitable. Moreover, pixel supervision often results in unstable training and poor generalization, especially when visual conditions differ significantly from training data. Addressing these limitations requires a paradigm shift toward more robust, noise-tolerant transfer mechanisms that do not rely on pixel-level fidelity.

The core innovation is the introduction of a latent-space post-training framework that aligns WM dynamics with robot action representations via contrastive learning. Unlike prior pixel-based methods, this approach constructs a shared low-dimensional video-action latent space, effectively capturing dynamic behaviors while avoiding visual noise. It employs lightweight adapters and a residual policy that refines actions based on WM-imagined dynamics, leaving the backbone fixed. This design simplifies training, enhances robustness, and improves transferability. The method also leverages synthetic imagined trajectories for effective knowledge transfer, enabling better generalization across tasks and environments. This shift from pixel supervision to latent alignment represents a fundamental advance in embodied AI training paradigms.

• �� Pretrain Cosmos-Predict2 structured video VAE to encode visual sequences into latent trajectories.
• �� Stage 1: Train video and action adapters (Bv, Ba) using reconstruction loss and contrastive InfoNCE loss to align WM latent dynamics with action embeddings.
• �� Use chunk-aware temporal contrast to preserve fine-grained dynamics.
• �� Stage 2: Freeze the VLA backbone; train a lightweight residual policy (fθ) that refines action latents by combining base actions with WM-guided residuals.
• �� Generate synthetic trajectories via WM for multiple environments; optimize the residual policy by contrastive alignment between WM latent trajectories and refined action latents.
• �� Decode the final action latents through a fixed decoder Da, and execute in simulation or real robot, iteratively improving alignment.

Experiments involve three benchmarks: RoboCasa, LIBERO, and Bridge-SIMPLER, with success rate as primary metric. Using approximately 3000 synthetic imagined trajectories, the method is compared against baselines like DreamGen and VLA-RFT. Ablation studies analyze the impact of contrastive objectives, temporal chunk alignment, and residual policy. Real robot tests on Franka arm validate transfer to physical tasks such as object manipulation. Hyperparameters include a contrastive temperature τ=0.1, training for around 7-15 hours, and success measured over 50 trials per task. Results consistently show superior performance and robustness, especially in unseen tasks and noisy visual conditions.

World2Act achieves success rate increases of +2.5% on RoboCasa, +6.0% on LIBERO, and +1.4% on Bridge-SIMPLER, outperforming pixel supervision methods. In real robot experiments, success rates improve by 6.7%, demonstrating real-world applicability. Ablation results indicate that bidirectional contrastive learning and temporal chunk alignment are critical for performance gains. The residual policy converges faster and generalizes better to unseen tasks, with success correlating strongly with latent space similarity. These findings confirm that latent dynamic alignment enhances stability, transferability, and robustness of embodied policies.

This approach is suitable for autonomous robotic manipulation, especially in scenarios with visual noise or domain shifts. It reduces reliance on pixel annotations, lowering data labeling costs. The method can be integrated into existing robotic systems to improve task transfer and generalization. Long-term, it opens pathways for scalable, noise-robust embodied AI capable of learning from imperfect world models, facilitating deployment in dynamic, real-world environments such as warehouses, homes, and factories.

Dependence on high-quality structured video VAEs limits applicability in highly complex or novel environments. The current approach may struggle with extremely dynamic scenes where latent representations are insufficient. Computational costs, especially for synthetic trajectory generation, remain significant. Further, real-world generalization requires online adaptation mechanisms to handle domain shifts. Future work should explore multi-modal latent representations and online learning to address these challenges.