WorldRoamBench: An Open-World Benchmark for Long-Horizon Stability of Interactive World Models
Introduces WorldRoamBench, a benchmark for long-horizon stability of interactive world models, evaluating per-frame action accuracy, visual drift, physics, and memory over 600+ cases.
Key Findings
Methodology
This framework integrates four evaluation dimensions: per-frame action accuracy to detect hidden failures, segment-based visual drift metrics to capture non-monotonic quality degradation, controllability-gated physics tests for plausibility, and scene/subject memory assessments via point-cloud reconstruction and tracking-plus-VLM reasoning. It covers over 600 real-world scenarios, assessing models’ performance during continuous 10-60s interactions. The approach bypasses semantic scale disparities by focusing on per-frame metrics, enabling fair comparison across models. Visual drift metrics detect mid-sequence collapses often missed by start-end comparisons. Physics evaluation ensures models obey physical laws like collision and optical consistency. Memory evaluation uses a novel revisit protocol, decoupling action errors from memory degradation, providing a comprehensive long-term stability assessment.
Key Results
- The top-performing models achieve an average score of only 0.65 across all dimensions, indicating significant room for improvement in long-horizon stability.
- Per-frame action accuracy averages around 60%, while trajectory scores hover near 0.75, with visual quality deteriorating after hundreds of frames, revealing cumulative errors.
- Physical interaction tests show frequent violations such as obstacle passing through, and memory assessments reveal scene reconstruction biases, highlighting persistent challenges in maintaining scene coherence over time.
Significance
This benchmark addresses a critical gap in evaluating long-term interactive models, moving beyond short clips to assess sustained stability, physical plausibility, and memory fidelity. It provides a standardized, comprehensive platform for researchers to benchmark progress, fostering development of models capable of realistic, stable, and physically grounded interactions in open-world environments. Such advancements are vital for applications in virtual reality, robotics, and AI-driven simulation, where long-term reliability is essential.
Technical Contribution
The framework introduces a per-frame action metric that overcomes semantic disparities, a segment-based visual drift measure to detect mid-sequence collapses, a controllability-based physics evaluation ensuring physical plausibility, and a scene memory protocol based on point-cloud reconstruction and tracking. These innovations enable a multi-dimensional, long-horizon assessment that was previously lacking, setting a new standard for evaluating interactive world models.
Novelty
This is the first benchmark explicitly designed for long-horizon, open-world interactive stability, integrating four evaluation dimensions with over 600 real-world scenarios. Its combination of per-frame metrics, visual drift detection, physics plausibility, and memory assessment offers a holistic, scalable evaluation framework that surpasses prior short-term or limited-scenario benchmarks.
Limitations
- The evaluation depends on pre-trained pose estimation and point-cloud reconstruction models, which may introduce biases or inaccuracies, affecting the metrics.
- Support for dynamic multi-object interactions and complex scene changes remains limited, requiring future extensions.
- Long-term cumulative errors still pose challenges, and the computational cost of comprehensive evaluation is high, necessitating optimization.
Future Work
Future directions include integrating multi-modal assessments, improving scene understanding for dynamic environments, developing adaptive metrics for diverse applications, and enhancing models’ robustness for real-world deployment. Expanding the benchmark to include multi-agent interactions and multi-object scenarios will further push the field toward more realistic, reliable systems.
AI Executive Summary
The development of interactive world models (IWMs) has accelerated rapidly, promising immersive virtual environments and advanced robotic control. However, current evaluation methods primarily focus on short clips, typically lasting 5-10 seconds, which are insufficient to gauge models’ long-term stability. As models grow more complex, their ability to maintain scene coherence, obey physical laws, and remember previous states over extended interactions remains untested.
To bridge this gap, we introduce WorldRoamBench, a comprehensive benchmark designed to evaluate models during continuous 10-60 second interactions across diverse open-world scenarios. This benchmark assesses four critical dimensions: action following, visual quality, physics plausibility, and scene memory. Its core innovation lies in the per-frame action metric, which bypasses semantic disparities and reveals hidden failures; a segment-based visual drift measure that detects non-monotonic quality collapses; a controllability-gated physics evaluation that ensures physical laws are obeyed; and a novel scene revisit protocol that decouples action errors from memory degradation.
The evaluation covers over 600 real-world cases, including natural, urban, and indoor environments, with models tested in first- and third-person views. Results reveal that even state-of-the-art models only achieve moderate scores (~0.65), with significant deficiencies in action accuracy, visual stability, physical consistency, and memory fidelity. These findings highlight the challenges of long-horizon stability and the need for further research.
This work provides a vital tool for benchmarking and advancing IWMs, fostering the development of models capable of sustained, physically grounded, and memory-reliable interactions. Future efforts will focus on multi-modal integration, dynamic scene understanding, and real-world deployment, ultimately pushing the boundaries of AI-driven virtual environments and autonomous systems.
Deep Dive
Abstract
Despite rapid progress in interactive world models (IWMs), existing benchmarks evaluate action following only at trajectory level and ignore memory and interaction physics. We introduce WorldRoamBench, an open-world benchmark for long-horizon stability across four dimensions, each with tailored innovations: (i) Action: per-frame action metric bypassing cross-model semantic scale disparity and exposing failures hidden by trajectory; (ii) Vision: segment-based drift metric capturing non-monotonic mid-sequence collapse missed by start-vs-end comparisons; (iii) Physics: controllability-gated evaluation over mechanics, optics, and 3D consistency, scoring plausibility under faithful action execution; (iv) Memory: action-decoupled protocol evaluating scene memory via transition-localized 3D point-cloud reconstruction and subject memory via tracking-plus-VLM reasoning. The benchmark comprises 600+ test cases across Nature, Urban, and Indoor scenes in first/third-person views with WASD 10-60s continuous interaction. Evaluating 10+ open/closed-source models reveals none reliably satisfies all dimensions; even the best achieves only moderate scores. Advances on WorldRoamBench are steps toward IWMs that are stable, physically grounded, memory-faithful, and deployable in real-world applications.