Programmable World Model
Introduced Programmable World Model, achieving 94% Count Accuracy and 98% State Accuracy by decoupling state evolution from visual generation.
Key Findings
Methodology
The framework decouples world-state evolution from visual generation. Natural-language instructions are translated into executable programs defining entity states and rules. A lightweight engine maintains an explicit, persistent global state, including off-screen entities and non-visual attributes. State-augmented 3D Oriented Bounding Boxes (OBBs) serve as intermediate representations, which, combined with camera trajectories, generate spatiotemporal conditioning signals for a pretrained video renderer.
Key Results
- Achieved 94% Count Accuracy and 98% State Accuracy on the CombatStateBench benchmark, significantly outperforming existing methods.
- Supports coherent long-horizon video generation, addressing consistency issues in interactive video world models.
- Ablation studies confirmed the effectiveness of decoupling state evolution from generative rendering.
Significance
This work addresses the lack of persistent state management and programmable rules in current video world models, enabling the creation of programmable worlds. The method has broad implications for gaming, virtual reality, and simulation training, where complex interactions and persistent states are critical.
Technical Contribution
Introduced a novel framework that separates state evolution from rendering, using state-augmented 3D OBBs as intermediate representations. Developed CombatStateBench as a benchmark for programmable world models.
Novelty
This is the first framework to explicitly maintain persistent states via a lightweight engine while integrating with generative models for programmable rules, contrasting with implicit state prediction methods.
Limitations
- Relies on pretrained video models, potentially limiting generalization to open-domain scenarios.
- Generation of complex geometry and dynamics depends on the capabilities of the generative model.
Future Work
Future directions include exploring more complex rule definitions, efficient state representations, and extensions to multi-agent environments.
AI Executive Summary
Current video world models excel at generating realistic visual environments but lack mechanisms for persistent state management and programmable rules. The Programmable World Model addresses this by decoupling world-state evolution from visual generation. It translates natural-language instructions into executable programs that define entity states and interaction rules. A lightweight engine maintains an explicit, persistent global state, including off-screen entities and non-visual attributes. State-augmented 3D Oriented Bounding Boxes (OBBs) serve as intermediate representations, which, combined with camera trajectories, generate spatiotemporal conditioning signals for a pretrained video renderer.
On the CombatStateBench benchmark, the method achieved 94% Count Accuracy and 98% State Accuracy, outperforming existing approaches. This design enables users to create playable games with predefined mechanics, direct control over individual entities, and persistent world states throughout gameplay.
While the method offers significant advancements, its reliance on pretrained video models may limit generalization to open-domain scenarios. Future research could explore more complex rule definitions, efficient state representations, and multi-agent environments, paving the way for broader applications in gaming, virtual reality, and simulation training.
Deep Analysis
Background
Video world models have advanced significantly, enabling realistic and interactive visual environments. However, they often lack explicit global state management, making it difficult to support persistent multi-entity interactions and complex rules. Existing methods rely on implicit state predictions, which struggle with consistency over long interactions.
Core Problem
Current models lack persistent state management and programmable rules, leading to inconsistencies in generated visuals for complex scenarios. Additionally, users cannot directly control individual entities or define intricate interaction rules.
Innovation
The Programmable World Model introduces several innovations: 1) A lightweight engine explicitly maintains persistent global states; 2) State-augmented 3D Oriented Bounding Boxes (OBBs) serve as intermediate representations; 3) Natural-language instructions are converted into executable rules, enabling complex interactions.
Methodology
- �� Translate natural-language instructions into executable programs defining entity states and rules.
- �� Use a lightweight engine to maintain explicit global states, including off-screen entities and non-visual attributes.
- �� Employ state-augmented 3D OBBs as intermediate representations, combined with camera trajectories to generate spatiotemporal conditioning signals.
- �� Utilize a pretrained video model as a generative renderer to synthesize realistic visuals.
Experiments
Experiments were conducted on the CombatStateBench benchmark to evaluate long-term consistency and accuracy. The benchmark includes diverse combat scenarios, testing the model's performance in character counting and state transitions. Baselines include existing video world models, and ablation studies validate the effectiveness of decoupling state evolution from rendering.
Results
The model achieved 94% Count Accuracy and 98% State Accuracy on CombatStateBench, outperforming existing methods. Ablation studies confirmed that decoupling state evolution from rendering was critical to performance improvements.
Applications
The method is applicable to gaming, virtual reality, and simulation training, particularly in scenarios requiring complex interactions and persistent states.
Limitations & Outlook
The reliance on pretrained video models may limit generalization to open-domain settings. Additionally, the generation of complex geometry and dynamic details depends on the generative model's capabilities.
Abstract
Recent video world models generate increasingly realistic and interactive visual experiences, yet lack reliable mechanisms for maintaining persistent world state and enforcing programmable rules over extended interactions. We introduce Programmable World Model, a framework that decouples world-state evolution from visual observation generation. An agent translates natural-language instructions into executable programs that specify entity states and state-transition rules, enabling direct control over individual entities and their interactions. A lightweight engine executes these programs to update and maintain an explicit, persistent global world state, including off-screen entities and non-visual attributes. To connect world state with visual generation, we introduce state-augmented 3D oriented bounding boxes (OBBs) as an intermediate representation. This representation, together with the target camera trajectory, is deterministically compiled into pixel-aligned spatiotemporal conditioning signals for a pretrained video model serving as the generative renderer. This design allows users to create playable games with predefined mechanics, direct control over individual entities, and persistent world state throughout gameplay. We further introduce CombatStateBench, a benchmark for evaluating programmable world models. On CombatStateBench, our method achieves 94% Count Accuracy and 98% State Accuracy, substantially outperforming existing interactive video world models while supporting coherent long-horizon generation. These results demonstrate the effectiveness of separating explicit state evolution from generative rendering for building persistent, programmable worlds.