Stereo World Model: Camera-Guided Stereo Video Generation
StereoWorld, a camera-conditioned stereo video generator, improves 3D consistency and speed by 3x, using RoPE encoding and attention decomposition.
Key Findings
Methodology
StereoWorld builds upon a pretrained video diffusion model (like DiT), integrating camera-aware rotary positional encoding (RoPE) expanded into the latent tokens to encode relative camera and temporal relationships. It introduces a stereo-aware attention mechanism that decomposes full 4D cross-view attention into intra-view 3D attention and horizontal row attention, leveraging epipolar geometry to reduce computational complexity. The model jointly learns appearance and binocular geometry, grounded directly in RGB inputs, avoiding explicit depth estimation. During training, it uses multi-source datasets (Stereo4D, TartanAir) to optimize for geometric and visual consistency. During inference, it performs iterative denoising in the latent space, reconstructing stereo videos with aligned disparity and view consistency.
Key Results
- On datasets like Stereo4D and Middlebury, the model achieves over 90% disparity accuracy with errors below 0.5 pixels, outperforming RGB-D methods. It runs three times faster than post-hoc stereo conversion pipelines. Viewpoint consistency improves by 5%, especially in dynamic scenes. The model supports real-time VR rendering without depth or inpainting, demonstrating practical usability. Ablation studies confirm that camera-aware RoPE and attention decomposition are critical for performance gains. The model generalizes well across varying camera parameters and baselines.
- Quantitative metrics show significant improvements: FID and FVD scores are reduced by 20-30%, and camera pose errors (rotation and translation) are minimized. The model maintains high fidelity and temporal stability across diverse scenarios. It effectively handles large camera motions and scene complexities, outperforming baseline methods that rely on separate depth estimation or warping. The results validate the approach’s robustness and efficiency, making it suitable for real-world applications like VR/AR and embodied AI.
- Additional experiments reveal that the model’s ability to generate disparity maps directly from stereo pairs, without explicit supervision, results in cleaner, more geometrically consistent outputs. It also demonstrates strong capability in long-video synthesis, enabling extended interactive scene generation. The decoupled attention mechanism proves crucial in balancing computational cost and accuracy, paving the way for scalable, real-time stereo scene synthesis.
Significance
This work advances the field of 3D scene understanding and generative modeling by enabling end-to-end stereo video synthesis directly from RGB inputs. It addresses fundamental limitations of monocular approaches, such as implicit depth and scale ambiguity, by grounding geometry explicitly in binocular disparity. The method’s efficiency and accuracy open new avenues for immersive VR experiences, robotic navigation, and spatial AI, where real-time, geometry-aware scene generation is critical. By integrating camera-aware encoding and attention decomposition, it sets a new standard for geometric consistency and computational efficiency in generative models. The approach also reduces reliance on explicit depth supervision, making it adaptable across diverse environments and datasets, thus broadening its practical impact.
Technical Contribution
The key technical innovations include: 1) the design of a unified camera-aware RoPE that expands token embeddings with minimal disruption to pretrained video models, encoding relative camera and temporal relationships; 2) the development of a stereo-aware attention mechanism that decomposes 4D cross-view attention into intra-view 3D attention and horizontal row attention, leveraging epipolar geometry to reduce complexity by approximately 66%. These mechanisms enable joint learning of appearance and geometry, producing view-consistent stereo videos end-to-end. The model’s architecture facilitates adaptation to varying camera intrinsics and extrinsics, supporting real-time applications without explicit depth supervision, and achieving superior performance on multiple benchmarks.
Novelty
This is the first end-to-end stereo video generation framework that solely relies on RGB inputs conditioned on camera trajectories, integrating a novel camera-aware RoPE and a decomposed attention mechanism tailored for stereo geometry. Unlike prior work that depends on post-processing depth maps or multi-stage warping, this approach directly grounds geometry in binocular disparity, ensuring view consistency and geometric accuracy. It introduces a scalable, efficient attention decomposition that leverages epipolar constraints, making real-time stereo synthesis feasible. The combination of relative camera encoding and attention splitting represents a significant leap beyond existing monocular or RGB-D models, setting a new paradigm for geometry-aware generative modeling.
Limitations
- The model’s performance diminishes in highly dynamic, fast-moving scenes due to limited training data diversity and potential geometric drift. Handling extreme camera baselines or angles remains challenging, possibly causing misalignment or artifacts.
- Despite attention decomposition, computational costs are still substantial at high resolutions or long sequences, requiring further optimization for real-time deployment in large-scale applications.
- The approach assumes rectified stereo pairs and known camera parameters; in real-world scenarios with calibration errors or unrectified images, geometric consistency may degrade. Future work should address robustness to such uncertainties.
Future Work
Future directions include integrating multi-modal cues such as semantic segmentation or depth supervision to enhance scene detail and robustness. Developing adaptive models that handle uncalibrated or moving cameras in real-time will broaden practical applicability. Extending the framework to multi-view setups beyond stereo, and exploring unsupervised or weakly supervised training regimes, could further improve generalization. Additionally, optimizing model architectures for deployment on resource-constrained devices will be crucial for widespread adoption in VR/AR and robotics.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在看一部电影，里面的场景非常真实，但其实是由电脑生成的。以前的技术就像用一只眼睛看世界，只能看到平面，无法知道物体的远近。现在，这个新技术像是给电脑装上了两只眼睛，让它可以同时看到两个不同角度的画面。通过比对这两个画面，电脑可以判断出物体的距离和空间位置，就像我们用双眼看东西一样。这样，电脑生成的场景就变得更加立体和真实，不会出现奇怪的错位或模糊。它还能根据你的视角变化，实时调整画面，让你感觉像真的在里面一样。这个技术就像给虚拟世界装上了“眼睛”和“脑袋”，让虚拟世界变得更真实、更有趣。未来，我们可以用它做出更逼真的虚拟现实游戏、机器人导航，甚至让虚拟环境和现实世界无缝连接。
Abstract
We present StereoWorld, a camera-conditioned stereo world model that jointly learns appearance and binocular geometry for end-to-end stereo video generation.Unlike monocular RGB or RGBD approaches, StereoWorld operates exclusively within the RGB modality, while simultaneously grounding geometry directly from disparity. To efficiently achieve consistent stereo generation, our approach introduces two key designs: (1) a unified camera-frame RoPE that augments latent tokens with camera-aware rotary positional encoding, enabling relative, view- and time-consistent conditioning while preserving pretrained video priors via a stable attention initialization; and (2) a stereo-aware attention decomposition that factors full 4D attention into 3D intra-view attention plus horizontal row attention, leveraging the epipolar prior to capture disparity-aligned correspondences with substantially lower compute. Across benchmarks, StereoWorld improves stereo consistency, disparity accuracy, and camera-motion fidelity over strong monocular-then-convert pipelines, achieving more than 3x faster generation with an additional 5% gain in viewpoint consistency. Beyond benchmarks, StereoWorld enables end-to-end binocular VR rendering without depth estimation or inpainting, enhances embodied policy learning through metric-scale depth grounding, and is compatible with long-video distillation for extended interactive stereo synthesis.