PAVXploreRL: Physical-Action-Visual World Model Reinforcement Learning with Action Exploration
Proposes PAVXploreRL, a reward-driven RL framework optimizing physical plausibility, action adherence, and visual fidelity, improving world model robustness by 5.6%.
Key Findings
Methodology
This work builds on the pretrained latent model VJEPA-2, designing three reward types targeting physical consistency, action adherence, and visual fidelity. It combines ID trajectories with noise-driven OOD action exploration, enabling explicit PAV optimization without paired videos. The approach employs multi-step autoregressive prediction, static regularization, and reward-guided training to enhance generalization and physical understanding. The reward functions incorporate pixel similarity, VJEPA encoder metrics, and perceptual quality, evaluated on both ID and OOD data, facilitating robust policy evaluation and model refinement.
Key Results
- Across multiple robotic benchmarks, PAVXploreRL achieves an average gain of 5.6%, outperforming pretrained baselines in structural fidelity, motion alignment, and visual details. The method reduces overestimation bias in policy evaluation, leading to more reliable performance estimates.
- Ablation studies confirm the importance of VJEPA encoder and predictor rewards, static regularization, and visual fidelity metrics, each contributing significantly to overall improvements.
- Experiments demonstrate better physical plausibility, action adherence, and visual realism, validating the effectiveness of the reward design and exploration strategy.
Significance
This research advances the development of reliable, generalizable world models for embodied AI, addressing key limitations of pixel-based and ID-only approaches. By explicitly optimizing PAV targets and enabling OOD exploration, it enhances model robustness in complex, real-world scenarios. The framework supports safer, more efficient policy evaluation, paving the way for autonomous robots capable of operating reliably in diverse environments, thus bridging the gap between simulation and reality.
Technical Contribution
The paper introduces a unified reward framework that explicitly optimizes physical plausibility, action adherence, and visual fidelity within a latent world model. It innovates by integrating OOD action exploration without paired videos, leveraging the predictive capabilities of VJEPA-2 in the latent space. The multi-strategy reward fusion and multi-stage training process significantly improve model generalization, robustness, and physical reasoning, setting a new standard for reinforcement learning in embodied AI.
Novelty
This is the first work to explicitly incorporate PAV objectives into a latent world model via reward-driven RL, utilizing OOD action exploration without relying on paired videos. It diverges from traditional pixel-level reconstruction methods, offering a novel paradigm that combines physical reasoning, action fidelity, and perceptual quality in a unified framework, thus enabling better generalization and physical consistency.
Limitations
- The reward design depends heavily on the predictive accuracy of the latent model, which may falter in highly complex or dynamic scenarios, limiting real-world applicability.
- Training involves multi-stage fine-tuning and RL, incurring high computational costs, restricting real-time deployment.
- OOD exploration remains limited in extremely unpredictable environments, requiring further enhancement for robustness against unforeseen dynamics.
Future Work
Future directions include integrating more sophisticated physics engines and multi-modal data to improve physical reasoning, developing adaptive reward mechanisms for better exploration efficiency, and enabling online learning for rapid adaptation to new tasks and environments. Extending the framework to multi-agent scenarios and real-world deployment will further broaden its impact.
AI Executive Summary
In recent years, the quest for autonomous robots capable of understanding and navigating complex environments has driven significant advances in world modeling. Traditional approaches primarily relied on pixel-level reconstruction, which often failed to capture the underlying physics and generalize beyond successful demonstrations. These models struggled with out-of-distribution actions, limiting their robustness and applicability in real-world scenarios.
This paper introduces PAVXploreRL, a reinforcement learning framework built upon a pretrained latent world model, VJEPA-2. The core idea is to explicitly optimize three critical targets: physical plausibility, action adherence, and visual fidelity. To achieve this, the authors design a reward system that leverages both in-distribution trajectories and noise-driven out-of-distribution actions, without requiring paired videos. This approach allows the model to explore and learn from a broader action space, improving its ability to generalize and accurately predict future states.
The technical innovation lies in combining multi-faceted rewards—pixel similarity, high-level perceptual metrics, and latent space predictions—within a multi-stage training pipeline. The model is fine-tuned with supervised learning and further refined through reinforcement learning, which guides it to produce physically consistent, controllable, and visually realistic predictions. Extensive experiments on robotic datasets demonstrate a 5.6% average improvement over baseline models, with qualitative results showing enhanced physical reasoning and action fidelity.
Beyond performance gains, this work significantly impacts embodied AI by providing a more reliable and generalizable world model. It reduces overestimation biases in policy evaluation, enabling safer and more accurate decision-making. The framework's ability to explore OOD actions without paired data opens new avenues for scalable, data-efficient reinforcement learning in robotics. Future research will focus on integrating richer physics, multi-modal inputs, and online adaptation, pushing the boundaries of autonomous embodied intelligence.
Deep Dive
Abstract
Action-conditioned world models are a key component of embodied AI, serving as scalable policy evaluators that reduce reliance on expensive real-world rollouts. To accurately capture diverse action-induced dynamics, such models should satisfy three key objectives-Physical Plausibility (P), Action Adherence (A), and Visual Fidelity (V), collectively referred to as PAV-while remaining robust to both in-distribution (ID) expert demonstrations and out-of-distribution (OOD) actions. However, existing methods primarily rely on ID action-video pairs and pixel-level reconstruction losses, which do not explicitly optimize PAV objectives and generalize poorly beyond expert data. To address this, we propose PAVXploreRL, a reinforcement learning framework built on a pretrained latent world model that explicitly optimizes PAV objectives through reward-driven training. To improve action generalization, our method jointly leverages ID trajectories and noise-driven OOD action exploration, without paired video supervision. Experiments show that PAVXploreRL consistently outperforms pretrained baselines, achieving a 5.6% average gain across benchmarks and producing higher-quality PAV properties. As a policy evaluator, it also yields more reliable performance estimates and reduces the overestimation bias of prior expert-only world models such as Ctrl-World. Code: https://github.com/Social-AI-Studio/PAVXploreRL