StateFlow: Building, Evolving, and Accessing 3D World States for Previsualization
StateFlow employs a structured 3D world state model with three stages—construction, evolution, and access—to enable high-quality, controllable previsualization for film and game design.
Key Findings
Methodology
StateFlow introduces a novel framework centered on a persistent, structured 3D scene state that encapsulates scene elements, geometry, attributes, and camera configurations. The process involves three key stages: (1) State Construction, which employs prior-guided dual-view initialization—combining semantic front views and spatial bird’s-eye views (BEV)—to generate a coherent initial scene; (2) State Evolution, where an intent-guided structured transition model (PlanVLM) updates scene content based on user commands, supporting scene expansion, style changes, and object motions without full scene regeneration; (3) State Access, which integrates render-feedback reflection to optimize camera trajectories, ensuring feasible, visually consistent viewpoints. The framework leverages pre-trained models like Gemini 3.1 for language understanding and Nano Banana 2 for high-fidelity video synthesis, enabling seamless conversion from natural language prompts to dynamic 3D worlds.
Key Results
- On the VBench dataset, StateFlow achieved an average score of 0.8484 across metrics such as subject consistency, background coherence, aesthetic quality, and motion smoothness, outperforming baseline methods like Animaker (0.8092) and MovieAgent (0.8075).
- The framework demonstrated robust scene editing capabilities, supporting scene expansion, style transfer, and object motion editing, with minimal scene reinitialization, thus significantly improving iterative workflow efficiency.
- Camera path optimization, guided by render-feedback, resulted in trajectories with higher visibility, collision avoidance, and motion smoothness scores, enhancing the realism and controllability of generated videos.
Significance
This work advances the field of previsualization by shifting from static, one-shot content generation to an interactive, editable 3D scene paradigm. It addresses longstanding challenges of scene consistency, iterative editing, and multi-view control, making virtual scene creation more accessible and flexible. The integration of multi-modal models and a structured state approach paves the way for automated, high-fidelity virtual production, with broad implications for filmmaking, game development, and architectural visualization. By enabling persistent scene states that can be continuously refined, StateFlow significantly reduces manual effort and enhances creative freedom, setting a new standard for dynamic virtual scene management.
Technical Contribution
The core technical innovation lies in the formulation of a persistent, structured 3D scene state that integrates multi-view cues and semantic priors, enabling conflict-aware initialization and intent-guided updates. The dual-view initialization combines semantic front views with spatial BEV layouts, resolving conflicts via prior-guided reasoning. The intent-guided structured transition model (PlanVLM) performs category-aware, localized updates—such as pose adjustments, object substitutions, and style modifications—without full scene regeneration. The render-feedback reflection mechanism couples visual rendering with geometric reasoning to optimize camera trajectories, ensuring feasible and visually appealing viewpoints. These components collectively enable a flexible, interactive, and high-fidelity previsualization pipeline that surpasses existing static or one-shot generative methods.
Novelty
This research is the first to formalize previsualization as a persistent, editable 3D world state rather than a sequence of static images or videos. Its key novelty includes the integration of multi-view conflict resolution during scene initialization, the use of intent-guided structured state transitions for scene evolution, and the coupling of render-feedback with camera planning. Unlike prior works that generate isolated assets or static scenes, StateFlow maintains a continuous, manipulable scene representation, enabling seamless iterative editing and multi-view consistency. This holistic approach bridges the gap between content generation and interactive scene management, marking a significant step forward in virtual scene previsualization.
Limitations
- The current framework faces challenges in handling highly complex or large-scale scenes, where scene details may degrade, or physical consistency may be compromised due to model limitations and computational constraints.
- Dependence on pre-trained models like Gemini 3.1 and Nano Banana 2 may limit adaptability across diverse domains or specialized content, affecting generalization.
- Real-time interaction and high-fidelity rendering at large scales remain computationally intensive, requiring further optimization for deployment in interactive applications such as VR/AR environments.
Future Work
Future directions include integrating physics-based simulation for more realistic object interactions, enhancing multi-modal reasoning for better scene understanding, and developing more efficient algorithms for real-time editing and rendering. Extending the framework to support larger, more dynamic scenes with non-rigid deformations and complex lighting conditions is also a priority. Additionally, exploring user-in-the-loop systems for more intuitive control and expanding applications to virtual production pipelines will further broaden the impact of this technology.
AI Executive Summary
Previsualization serves as a vital bridge between initial creative ideas and final production in film, gaming, and architecture. Traditional methods, often relying on static rendering or one-shot generative models, struggle to support iterative editing, multi-view control, and scene consistency. These limitations hinder rapid prototyping and creative exploration, especially in complex scenes requiring frequent adjustments.
Yuyang Yin and colleagues address these challenges with StateFlow, a pioneering framework that redefines previsualization through a persistent, structured 3D scene state. Unlike conventional approaches, which treat each output as isolated, StateFlow maintains a continuous scene representation that can be constructed, evolved, and accessed interactively. This approach enables creators to modify scene elements, adjust camera paths, and refine visual styles seamlessly, fostering a more dynamic and flexible workflow.
The framework operates in three interconnected stages. First, State Construction employs prior-guided dual-view initialization—combining semantic front views with spatial BEV layouts—to generate an initial, conflict-resolved 3D scene. This step leverages vision-language models like Gemini 3.1 and geometric priors to ensure scene coherence. Next, State Evolution uses an intent-guided structured transition model, PlanVLM, to update scene content based on user commands, supporting scene expansion, style changes, and object motions without needing to regenerate the entire scene. This incremental update mechanism preserves scene memory and continuity, enabling iterative editing.
Finally, State Access integrates render-feedback reflection to optimize camera trajectories, ensuring viewpoints are physically feasible, visually appealing, and aligned with creative goals. This feedback loop combines geometric reasoning with visual rendering, reducing issues like occlusion or unnatural camera movements. The entire pipeline is supported by pre-trained models, allowing efficient conversion from natural language prompts to dynamic 3D worlds.
Experimental results demonstrate that StateFlow outperforms existing methods like Animaker and MovieAgent in detail preservation, motion smoothness, and editing flexibility. Its ability to support complex scene editing, style transfer, and multi-view visualization makes it highly suitable for cinematic storyboarding, game prototyping, and architectural visualization. The framework’s modular design and reliance on pre-trained models also facilitate broad adaptability across domains.
Looking ahead, the authors plan to incorporate physics-based simulation and multi-modal reasoning to handle more complex scenes and interactions. They aim to optimize computational efficiency for real-time applications, such as virtual reality and interactive design. Overall, StateFlow represents a significant leap toward intelligent, interactive, and high-fidelity virtual scene management, promising to transform how digital content is created and manipulated in the future.
Deep Dive
Glossary
Structured 3D Scene State (结构化3D场景状态)
一种包含场景元素、几何、属性和相机配置的持久化模型，用于支持多次编辑和动态演变。
作为StateFlow的核心，确保场景在多次修改中保持一致性和可控性。
Dual-View Initialization (双视角初始化)
结合前视图和鸟瞰图信息，通过先验引导解决多视角冲突，生成一致的场景布局。
用于场景构建阶段，确保场景元素在不同视角下的合理性。
Intent-Guided Structured Transition (意图引导的结构化状态转移)
基于用户自然语言指令，利用模型预测场景元素的局部变化，实现连续演变。
支持场景扩展、风格变换和对象运动，避免频繁重建场景。
Render-Feedback Reflection (渲染反馈反思)
结合渲染结果和几何信息，优化摄像机路径，确保视角合理、运动自然。
提升摄像机轨迹的可行性和视觉质量。
PlanVLM (计划视觉语言模型)
基于视觉语言模型的场景状态转移预测器，生成场景更新的动作计划。
在状态演化阶段指导场景的连续调整。
Nano Banana 2 (纳米香蕉2)
高质量视频生成模型，用于将场景几何和属性升维为表达丰富的视频内容。
实现从静态场景到动态视频的高效转换。
Gemini 3.1 (双子座3.1)
先进的视觉语言模型，用于场景理解、意图解析和布局推理。
支撑StateFlow的自然语言到场景的转换。
Conflict-Aware Initialization (冲突感知初始化)
检测并解决多视角生成内容中的冲突，确保场景布局合理。
在状态构建阶段应用，提升场景一致性。
Open Questions Unanswered questions from this research
- 1 尽管StateFlow在静态场景和简单动态场景中表现优异，但在复杂动态场景（如非刚性变形、多主体交互）中的表现仍需验证。未来需要结合物理模拟和学习增强机制，以提升真实性和复杂度。
- 2 模型对多模态信息的融合依赖预训练模型，泛化能力在特定专业或极端场景下可能不足，需探索领域适应和微调策略。
- 3 实时交互和高帧率场景的处理仍存在计算瓶颈，未来需优化算法效率和硬件适配，以支持更大规模和更复杂的应用。
- 4 场景的多尺度、多层次细节管理尚待完善，尤其是在大规模城市或复杂机械场景中，如何平衡细节丰富度与计算成本是未来的关键。
- 5 用户交互界面设计和操作流程仍需优化，以实现更直观、自然的场景编辑体验。
Applications
Immediate Applications
电影预可视化
利用StateFlow快速构建和编辑电影场景，支持故事板设计、镜头规划和动态特效预览，显著缩短制作周期，降低专业门槛。
游戏场景原型设计
为游戏开发者提供交互式场景编辑工具，支持多角度预览和动态演变，提升原型迭代效率，助力创意实现。
建筑虚拟漫游
结合自然语言描述快速生成建筑模型，支持多视角观察和场景调整，辅助设计决策和客户演示。
Long-term Vision
虚拟现实内容生成
实现沉浸式虚拟环境的自动化生成与实时编辑，推动虚拟旅游、虚拟培训等应用的发展。
自动化虚拟制作平台
结合AI、物理模拟和多模态感知，打造全自动化的虚拟内容生产链，降低内容创作成本，提升个性化定制能力。
Abstract
Previsualization is an intermediate layer between ideas and production in film, games, architecture, and urban design. It lets creators iteratively refine scenes, actions, cameras, and spatial-temporal dynamics. Yet existing generative methods rely on simple prompts to jointly control all of these factors through one-shot image or video synthesis, offering weak controllability and limited support for iterative editing. Fundamentally, a world comprises multiple elements with geometry, appearance, and other attributes, together with cameras. Different frames are produced through local modifications or recombinations of this shared state, which is otherwise largely reused. Therefore, we argue that the missing component is an explicit and persistent working state. To address this, we present StateFlow, a state-centric framework for generative previsualization. Rather than generating videos in one shot, StateFlow uses an editable 3D world to organize scene structure, evolution, and cameras, while off-the-shelf video models enhance visual quality when higher fidelity is desired. This world is maintained as a persistent structured 3D state of scene elements and camera configurations, serving as the core working representation for previsualization. Built on this insight, StateFlow has three stages to construct, evolve, and access the world state. State construction lifts generated 2D content into a coherent 3D world through prior-guided, conflict-aware dual-view initialization, while State evolution translates user intent into structured state transitions while preserving world memory, avoiding full-scene regeneration for each edit. State access uses render-feedback reflection to refine camera plans into visually feasible trajectories, avoiding reliance on VLM semantics alone. Experiments show that StateFlow produces high-quality 3D worlds for video creation and game-like prototyping.
References (20)
Seedance 2.0: Advancing Video Generation for World Complexity
Team Seedance, Deying Chen, Liyang Chen et al.
Genie: Generative Interactive Environments
Jake Bruce, Michael Dennis, Ashley Edwards et al.
A Collaborative Previsualization Tool for Filmmaking in Virtual Reality
Dui Ardal, Simon Alexandersson, Mirko Lempert et al.
WorldStereo: Bridging Camera-Guided Video Generation and Scene Reconstruction via 3D Geometric Memories
Yisu Zhang, Chenjie Cao, Tengfei Wang et al.
MMGDreamer: Mixed-Modality Graph for Geometry-Controllable 3D Indoor Scene Generation
Zhifei Yang, Keyang Lu, Chao Zhang et al.
Infinite Photorealistic Worlds Using Procedural Generation
Alexander R. E. Raistrick, Lahav Lipson, Zeyu Ma et al.
PanoWorld-X: Generating Explorable Panoramic Worlds via Sphere-Aware Video Diffusion
Yuyang Yin, Hao Guo, Fangfu Liu et al.
TiP4GEN: Text to Immersive Panorama 4D Scene Generation
Ke Xing, Hanwen Liang, Dejia Xu et al.
WorldGen: From Text to Traversable and Interactive 3D Worlds
Dilin Wang, Hyunyoung Jung, Tom Monnier et al.
From Camera-Eye to AI: Exploring the Interplay of Cinematography and Computational Visual Storytelling
Brett A. Halperin, Stephanie M. Lukin
High-Resolution Image Synthesis with Latent Diffusion Models
Robin Rombach, A. Blattmann, Dominik Lorenz et al.
DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation
Jiaxiang Tang, Jiawei Ren, Hang Zhou et al.
Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation
Zibo Zhao, Zeqiang Lai, Qin Lin et al.
WorldSimBench: Towards Video Generation Models as World Simulators
Yiran Qin, Zhelun Shi, Jiwen Yu et al.
Camera Control in Computer Graphics
M. Christie, P. Olivier, Jean-Marie Normand
WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling
Wenqiang Sun, Haiyu Zhang, Haoyuan Wang et al.
A Systematic Review of Architectural Design Collaboration in Immersive Virtual Environments
R. Yu, N. Gu, Gun Lee et al.
Experiments in the use of game technology for pre-visualization
Michael Nitsche
Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models
Hanwen Liang, Yuyang Yin, Dejia Xu et al.
Matrix-3D: Omnidirectional Explorable 3D World Generation
Zhongqi Yang, Wenhang Ge, Yuqi Li et al.