WorldMark: A Unified Benchmark Suite for Interactive Video World Models
Introduces WorldMark benchmark using model adapters and control-system metrics to evaluate interactive video world models' response dynamics.
Key Findings
Methodology
This work develops a unified action interface by translating a shared WASD+L/R vocabulary into each model’s native control format via per-model adapters. The benchmark includes 500 diverse scenes and 15 action trajectories, covering styles, viewpoints, and difficulty levels. Using control system principles, four key metrics—direction accuracy, purity, response latency, and motion stability—are computed separately for translation and rotation axes, providing a granular view of response behaviors. Additional world memory and visual quality metrics complement the analysis. All measurements are deterministic, leveraging feed-forward estimators to ensure reproducibility. This approach overcomes limitations of prior benchmarks that only measured trajectory error, enabling detailed insights into response dynamics.
Key Results
- Experimental results reveal a trade-off: models with faster responses tend to be less stable. Lyra 2.0 exhibits the lowest response latency (~0.15s) and high directional accuracy (~98%), but suffers from lower motion stability. HY-World 1.5 achieves top visual quality (score 94.17) but has higher latency (~0.35s). Fine-grained analysis shows models follow translation commands nearly perfectly but respond poorly to rotations. Stylized scenes challenge global consistency, yet responses remain largely unaffected. These findings demonstrate the importance of multi-dimensional metrics for comprehensive evaluation.
- Across 10 models, the proposed metrics effectively differentiate response speed, stability, and scene consistency. Models like Lyra 2.0 excel in responsiveness but lag in stability, while others like HY-World prioritize visual fidelity at the cost of latency. The detailed per-axis assessment uncovers specific weaknesses, guiding targeted improvements. The benchmark’s reproducibility and broad scene coverage make it a robust tool for advancing interactive world models, fostering fair comparisons and accelerating development.
- The results underscore that optimizing for one aspect (speed, stability, or quality) often compromises others. The detailed metrics expose nuanced behaviors, such as models that nearly perfectly follow translation commands but ignore rotation, or those that respond quickly but oscillate. These insights are crucial for designing balanced models suitable for real-world applications like VR, AR, and gaming, where responsiveness and stability are both vital. The benchmark thus provides a comprehensive framework for future research.
Significance
The WorldMark benchmark addresses a critical gap in evaluating interactive video world models by quantifying their dynamic response characteristics. Moving beyond traditional visual and memory metrics, it introduces a control-theoretic perspective, enabling detailed analysis of how models react to user commands. This systematic approach facilitates fair comparison across heterogeneous models, guiding targeted improvements. Its broad scene coverage and reproducibility make it a valuable tool for academia and industry, fostering innovation in virtual environments, gaming, and AR applications. The benchmark’s online platform also promotes community engagement, encouraging continuous progress in the field.
Technical Contribution
This work’s key technical innovation lies in integrating control system principles into model evaluation. The per-model adapters enable fair comparison despite heterogeneous control formats. The four action response metrics—direction accuracy, purity, latency, and stability—are derived from established control theory, adapted for visual data analysis via optical flow and feature matching. All metrics are deterministic, ensuring reproducibility. The comprehensive suite combines dynamic response assessment with scene memory and visual quality evaluation, forming a multi-dimensional performance profile. This systematic framework significantly advances the evaluation methodology for interactive world models, bridging the gap between theoretical control analysis and practical generative modeling.
Novelty
This is the first work to embed control system metrics into the evaluation of interactive video world models, moving beyond simple trajectory errors. The introduction of per-model adapters for heterogeneous control formats enables fair, multi-model comparison. The detailed, axis-resolved response metrics reveal behaviors previously hidden by aggregate scores, such as the speed-stability trade-off. This comprehensive, multi-faceted evaluation approach is a novel contribution, setting a new standard for benchmarking in the field and facilitating targeted model improvements.
Limitations
- The current metrics rely on optical flow and feature matching, which may be sensitive to scene complexity and visual artifacts, potentially affecting accuracy in challenging scenarios. Model adapters, while low-cost, may face difficulties in extreme control formats or highly specialized control schemes. The benchmark covers a wide but finite set of scenes and trajectories, leaving out some real-world complexities like long-term interactions and multi-modal inputs. Future work should incorporate robustness testing and extend to multi-modal evaluation.
Future Work
Future directions include integrating robustness and generalization metrics, expanding scene diversity, and incorporating long-term interaction assessments. Combining visual, auditory, and haptic modalities could provide a more holistic evaluation of interactive models. Developing adaptive adapters that learn optimal mappings could streamline model integration. Additionally, fostering community-driven benchmarks and challenges will accelerate progress, ultimately enabling more responsive, stable, and realistic virtual environments for industry and research.
AI Executive Summary
Interactive video world models have rapidly evolved, enabling virtual agents to respond dynamically to user commands within complex environments. However, existing benchmarks primarily focus on visual fidelity and scene consistency, neglecting the critical aspect of response dynamics—how quickly and stably models react to control inputs. This gap hampers progress, as models may excel in appearance but perform poorly in real-time interaction.
To address this, the authors introduce WorldMark, a comprehensive benchmark that leverages control system theory to evaluate models’ response behaviors. Central to this framework are per-model adapters that translate a shared action vocabulary into each model’s native control format, ensuring fair comparison despite heterogeneity. The benchmark measures four key response metrics—direction accuracy, purity, response latency, and motion stability—each evaluated separately for translation and rotation. These are complemented by world memory and visual quality assessments, creating a multi-dimensional performance profile.
The experimental results reveal nuanced trade-offs: models with rapid responses often exhibit decreased stability, while high visual quality models tend to respond more slowly. For example, Lyra 2.0 demonstrates the lowest latency (~0.15s) but suffers from lower stability, whereas HY-World 1.5 excels in visual fidelity but has higher latency (~0.35s). These findings highlight the importance of balanced optimization across multiple response facets.
Beyond advancing evaluation methodology, WorldMark’s broad scene coverage and reproducibility foster fair comparisons and targeted improvements. Its online platform encourages community engagement, driving the development of more responsive, stable, and realistic interactive models. Future work aims to incorporate robustness, multi-modal inputs, and long-term interaction assessments, pushing the field toward more practical applications in VR, AR, and gaming. Overall, this benchmark sets a new standard for systematic evaluation, catalyzing innovation in interactive virtual environments.
Deep Dive
Abstract
Unlike text- or image-driven video generation, an interactive world model is driven by actions: the user acts, and the world responds. Two obstacles stand in the way of fair and comprehensive evaluation. First, models take actions in incompatible formats---captions, camera trajectories, action functions---so no shared protocol has been established. Second, while existing benchmarks have advanced world memory and visual quality, action following is reduced to trajectory or direction error, which collapses a whole path into one number: not how quickly the world reacts to a command switch, nor how cleanly it moves along the commanded axis. WorldMark removes both obstacles. Per-model adapters translate a shared WASD-style vocabulary into each model's native control format, so ten heterogeneous models receive semantically identical instructions across 500 standardized cases spanning styles, viewpoints, and difficulty tiers; a new model costs one adapter. On this common ground we characterize action dynamics through a control-systems lens---direction accuracy, direction purity, response latency, and motion stability, each resolved per axis---alongside suites for world memory and visual quality. Together they expose differences existing protocols cannot see: the fastest responders are often the least stable, a trade-off no single action metric captures; per-axis resolution reveals models that follow translation almost perfectly while barely responding to rotation; the model with the best perceptual and aesthetic quality ranks last in translational direction accuracy and latency; and stylized scenes cost every model global consistency while leaving action dynamics largely intact. We will release all data, evaluation code, and model outputs.