Hierarchical Planning with Latent World Models
Hierarchical Planning with Latent World Models (HWM) enables zero-shot long-horizon visual control, achieving 70% success on real robot tasks, with 3× less computation.
Key Findings
Methodology
HWM employs multi-scale latent space world models trained solely via next-latent prediction, integrating a learned action encoder to compress primitive actions into macro-actions. The architecture involves a high-level long-horizon model and a low-level short-horizon model, which are coupled through latent matching without task-specific rewards or policies. During planning, the high-level model generates subgoals that guide the low-level controller, enabling efficient long-term planning in visual environments. The training process relies on autoregressive latent predictions, ensuring end-to-end differentiability and scalability across diverse tasks.
Key Results
- In real-world Franka robot pick-and-place tasks, HWM increased success rate from 0% (single-level planner) to 70%, demonstrating its ability to handle non-greedy, multi-stage tasks. In simulated push manipulation, success rates improved by 44%, reaching 61% at 75 steps, outperforming baselines. In maze navigation, success rates increased from 44% to 83% in out-of-distribution environments, with up to 3× reduction in planning compute, validating the efficiency gains. Across architectures like VJEPA2-AC, DINO-WM, and PLDM, HWM consistently enhanced long-horizon performance.
- The experiments highlight HWM’s capacity for zero-shot generalization, robustness in complex tasks, and computational efficiency, marking a significant step toward autonomous agents capable of complex visual reasoning and manipulation without task-specific training.
Significance
This work addresses fundamental limitations in visual world model planning, notably the exponential growth of search space and error accumulation in long-horizon tasks. By introducing a hierarchical framework that operates entirely in learned latent spaces, HWM enables zero-shot, non-greedy control in real robots and simulated environments. Its ability to generate subgoals dynamically and compress action sequences into macro-actions reduces computational costs while improving success rates. This approach paves the way for scalable, generalizable autonomous systems capable of complex multi-stage reasoning, bridging the gap between high-dimensional perception and long-term decision-making. The methodology’s flexibility and efficiency hold promise for advancing robotics, autonomous vehicles, and AI planning in diverse real-world scenarios.
Technical Contribution
HWM innovates by integrating multi-scale latent space models with a learned macro-action encoder, enabling hierarchical model predictive control directly on visual inputs. Unlike prior methods relying on low-dimensional states or task-specific rewards, HWM’s architecture allows end-to-end training solely via next-latent prediction, facilitating zero-shot generalization. Its key technical advances include the coupling of long- and short-horizon models through latent matching, and the compression of primitive actions into macro-actions, significantly reducing search complexity. This design provides theoretical guarantees of long-term planning accuracy and computational efficiency, opening new avenues for scalable, model-based visual control.
Novelty
This research is the first to demonstrate hierarchical model predictive control directly on learned visual world models without task-specific rewards or policies. The core innovation lies in multi-scale latent space modeling combined with a learned macro-action encoder, enabling efficient long-horizon planning. Unlike existing hierarchical RL or MPC approaches limited to low-dimensional states, HWM operates end-to-end on pixel-based observations, achieving zero-shot generalization to complex, multi-stage tasks. Its ability to dynamically generate subgoals and compress action sequences distinguishes it as a pioneering framework for scalable, visual-based autonomous control.
Limitations
- Despite its robustness, HWM’s performance can be affected by perceptual inaccuracies such as depth errors, which may impair subgoal precision. Long-term prediction errors still accumulate over extended horizons, limiting success in highly complex environments. The training process requires substantial offline data and computational resources, which may hinder deployment in resource-constrained settings. Additionally, the current macro-action encoder may need adaptation for tasks with highly variable action durations or complex dynamics, indicating avenues for further refinement.
Future Work
Future research will focus on enhancing perceptual robustness, integrating multi-modal sensory data, and developing adaptive macro-action encoders for variable-duration tasks. Exploring online learning and continual adaptation could further improve real-time performance. Extending the framework to multi-agent scenarios and more diverse environments will test its scalability. Additionally, combining HWM with reinforcement learning techniques may enable autonomous policy refinement, pushing toward fully autonomous, general-purpose visual agents capable of long-term reasoning and manipulation.
AI Executive Summary
Long-horizon planning in high-dimensional visual environments remains a central challenge in robotics and AI. Traditional approaches often struggle with the exponential growth of search space and the accumulation of prediction errors, especially when dealing with complex, multi-stage tasks. Existing hierarchical methods either rely on task-specific policies or low-dimensional state assumptions, limiting their generalization and scalability.
This paper introduces Hierarchical Planning with Latent World Models (HWM), a novel framework that operates entirely within learned latent spaces. By employing multi-scale world models trained solely through next-latent prediction, HWM captures long-term dynamics efficiently. The architecture couples a high-level long-horizon model with a low-level short-horizon model via latent matching, enabling the generation of subgoals without task-specific rewards or policies. To further improve efficiency, a learned macro-action encoder compresses primitive actions, reducing the search space and computational load.
Experimental results demonstrate HWM’s effectiveness across real-world and simulated tasks. In Franka robot pick-and-place, success rates soared from 0% to 70%, showcasing its ability to handle non-greedy, multi-stage behaviors. In push manipulation and maze navigation, success rates increased by up to 44% and 39%, respectively, with up to 3× less planning compute. These findings confirm that hierarchical latent space planning significantly enhances long-term control, generalization, and efficiency.
Overall, HWM represents a major advance in visual model predictive control, enabling zero-shot, non-greedy manipulation and navigation in complex environments. While challenges like perceptual errors and long-term prediction inaccuracies remain, ongoing work aims to improve robustness and scalability, paving the way for autonomous agents capable of sophisticated, long-term reasoning in diverse real-world scenarios.
Deep Dive
Abstract
World models are a promising path to zero-shot embodied control through planning. However, existing world model planners struggle on long-horizon, multi-stage tasks: prediction errors compound and naive search is exponential in the planning horizon. Hierarchy mitigates both by decomposing tasks into shorter, tractable subproblems; yet prior hierarchical approaches either amortize control into task-specific policies (hierarchical RL) or assume low-dimensional states and known dynamics (classical hierarchical MPC). We present Hierarchical Planning with Latent World Models (HWM), an architecture and planning paradigm for hierarchical model predictive control (MPC) directly on visual world models trained solely via next-latent prediction. HWM learns world models at multiple temporal scales within a shared latent space, so predictions from the long-horizon model serve as subgoals for the short-horizon model via latent matching, without task-specific rewards, skill learning, or hierarchical policies. To keep long-horizon search tractable, HWM learns an action encoder that compresses primitive action chunks into latent macro-actions. On real-world Franka manipulation, HWM solves pick-and-place from a single goal image at 70% success vs. 0% for single-level planning. Across simulated push manipulation and maze navigation, HWM consistently improves performance on long-horizon tasks while requiring up to 3x less planning compute.