ReWorld: An Interactive World Model with Long-Horizon Memory
ReWorld uses split training and pose-indexed landmark bank to balance long-term memory and real-time control, achieving state-of-the-art fidelity.
Key Findings
Methodology
ReWorld employs a dual training strategy, separating control and memory via mixed attention windows and random head routing, enabling capabilities to develop independently. During training, chunk-drop augmentation enhances robustness to sparse memory. At inference, a pose-indexed landmark bank combined with a bounded KV cache supports long-horizon scene recall. Distribution-matching distillation with LoRA adapters compresses multi-step generation into four denoising steps, allowing high-fidelity, real-time streaming of 704×1280 videos across diverse visual styles. Multi-source data pipelines unify synthetic, real, and game footage on a consistent physical scale, ensuring accurate action correspondence. The system outperforms six recent models on control fidelity (rotation error 11.95°) and video quality, especially in long rollouts where full attention memory is exhausted.
Key Results
- ReWorld achieves a rotation error of 11.95°, surpassing all comparison models; in 64s rollouts, its fixed 12-chunk cache still reconstructs the initial view, demonstrating robust long-term memory. It generates high-resolution videos across styles, maintaining real-time performance. The pose-indexed landmark bank and chunk-drop training enable effective long-horizon scene recall within limited cache budgets, validated through extensive ablation studies.
- The model’s ability to revisit and accurately regenerate scenes after long intervals is confirmed by quantitative metrics and visual assessments. Its control accuracy and scene consistency outperform baselines, especially in complex, multi-source environments. The distillation approach ensures the model remains lightweight and fast, suitable for deployment in interactive applications.
- Ablation experiments show that the combination of pose-based retrieval, mixed attention windows, and sparse cache training is critical for balancing control and memory. The approach generalizes well across synthetic, real, and game data, indicating strong robustness and transferability.
Significance
This work addresses a fundamental challenge in interactive world modeling: reconciling the need for long-term scene memory with the demands of real-time control. By decoupling training and inference, and leveraging pose-aware edge storage, ReWorld sets a new benchmark for scene continuity and responsiveness. Its multi-source data pipeline and efficient distillation enable deployment in realistic scenarios like VR, autonomous navigation, and game development. The approach paves the way for more intelligent, persistent virtual environments, bridging the gap between static scene understanding and dynamic interaction.
Technical Contribution
ReWorld introduces a novel framework combining split attention training with pose-indexed landmark retrieval, effectively decoupling control and memory capabilities. The mixed per-head attention and random head routing ensure capability independence, while chunk-drop augmentation aligns training with sparse inference caches. The pose-aware landmark bank, managed via a redundancy-based eviction policy, supports unbounded scene recall within fixed memory constraints. Distribution-matching distillation with LoRA adapters compresses multi-step sampling into four denoising steps, enabling real-time streaming. These innovations collectively advance the state-of-the-art in long-horizon, high-fidelity interactive scene generation.
Novelty
This paper is the first to explicitly decouple control and long-term memory training via mixed attention windows and random head routing, enabling models to learn both abilities without mutual interference. The pose-indexed landmark bank, combined with sparse chunk-drop training, provides a scalable long-term memory solution within fixed cache budgets. The integration of distribution-matching distillation with LoRA adapters for multi-source data is a significant engineering breakthrough, allowing high-quality, real-time video generation across diverse environments. These innovations collectively push the frontier of interactive scene modeling.
Limitations
- Despite its robustness, ReWorld may struggle with highly dynamic or unpredictable environments where scene changes exceed stored landmarks or memory capacity, leading to potential scene inconsistencies.
- The pose-based landmark retrieval relies on accurate pose estimation; errors in pose alignment could degrade recall fidelity, especially in real-world scenarios with noisy sensors.
- The current system’s computational demands, especially during training and distillation, may limit scalability or deployment on resource-constrained devices, requiring further optimization.
Future Work
Future research will focus on adaptive memory management strategies, integrating reinforcement learning to optimize landmark selection dynamically. Enhancing pose estimation robustness and extending multi-modal inputs (audio, haptic) could further improve scene understanding. Additionally, scaling the system for larger environments and more complex interactions remains a key goal, aiming for broader applicability in autonomous systems and immersive virtual worlds.
AI Executive Summary
ReWorld introduces a groundbreaking approach to interactive world modeling by effectively balancing long-term scene memory with real-time control. Traditional models often face a trade-off: short windows enable quick responses but limit scene recall, while long-term memory mechanisms tend to slow down or degrade control fidelity. To overcome this, ReWorld employs a novel training paradigm that separates control and memory learning, utilizing mixed attention windows and random head routing to ensure capabilities develop independently. During inference, a pose-indexed landmark bank, combined with a fixed-size KV cache, supports unbounded scene recall, enabling the model to revisit distant locations accurately.
The core innovation lies in integrating pose-aware edge storage with a sparse chunk-drop training regime, which prepares the model to operate reliably under limited memory budgets. This is complemented by a distribution-matching distillation process using LoRA adapters, compressing multi-step generation into just four denoising steps, thus achieving real-time streaming of high-resolution videos across diverse visual styles. Extensive experiments demonstrate that ReWorld outperforms six recent models in control fidelity, scene recall, and video quality, especially in long-horizon rollouts where traditional full attention models fail due to memory constraints.
This work significantly advances the field of virtual environment simulation, autonomous navigation, and immersive VR, providing a scalable, robust, and high-fidelity solution for long-term scene understanding and interaction. Its ability to unify synthetic, real, and game data on a common physical scale ensures broad applicability. While challenges remain in dynamic environments and pose estimation accuracy, the proposed framework opens new avenues for persistent, intelligent virtual worlds, paving the way for future innovations in AI-driven scene modeling and interaction.
Deep Dive
Abstract
An interactive world model must follow the user's actions, remember the places it has shown, and stream in real time. The tension is structural: control wants a short horizon, memory wants an unbounded one. ReWorld separates the two during training and bounds them at inference. Mixed per-head attention windows confine most heads to the recent past while a small set of global heads attends over the entire history, and random head routing keeps either capability from binding to particular heads; random chunk dropping makes sparse histories in-distribution. At inference the whole past lives under a fixed budget: a bounded KV cache backed by a pose-indexed landmark bank, from which the model retrieves the landmarks nearest the current pose. A metric-scale-aligned data engine places eight sources -- Unreal-rendered fly-throughs, game roaming, and real-world footage -- on one physical action scale, so the same key press moves the camera the same distance in every source, and palindrome trajectories supply the revisit evidence that memory training needs. Distribution-matching distillation confined to a LoRA adapter then compresses sampling to four steps: one backbone serves both a high-fidelity multi-step mode and a real-time interactive one, streaming 704x1280 video across photorealistic, game-style, and stylized worlds. Under a three-axis protocol covering action following, long-horizon recall, and video quality, against six recent interactive world models it attains the best control fidelity ($11.95^\circ$ rotation error and the best camera-motion consistency) and the best generation quality; and on minute-long out-and-back rollouts ($64$\,s, $384$ latents), its fixed 12-chunk cache still regenerates the starting view -- at rollout lengths where a sliding window has long evicted the evidence and full-KV attention runs out of memory.