VGGT-World: Transforming VGGT into an Autoregressive Geometry World Model

Recent advances in 3D scene understanding have been driven by Geometry Foundation Models like VGGT, which can recover scene structure from monocular images. These models excel at static scene analysis but lack temporal prediction capabilities. Video generation methods, such as Cosmos and Wan, attempt to model scene evolution by generating future frames, yet they often produce geometrically inconsistent results due to their focus on photometric details. Efforts to incorporate geometric priors into video models, like Geometry Forcing and GeoWorld, improve structural stability but are computationally expensive and complex. Meanwhile, latent-space models like DINO-Foresight leverage semantic features for prediction but lack explicit geometric encoding. This landscape underscores the need for a geometry-centric predictive framework that balances efficiency, accuracy, and geometric consistency.

The main challenge is how to perform stable, accurate scene geometry forecasting directly in high-dimensional feature spaces derived from pre-trained GFMs. Velocity prediction methods in flow matching often fail due to the curse of dimensionality, leading to noisy and unstable trajectories. Autoregressive rollout suffers from exposure bias, where errors compound over long horizons, degrading prediction quality. Existing approaches either reduce dimensionality at the cost of losing geometric fidelity or rely on heavy video generation pipelines that are computationally prohibitive. Therefore, developing a stable, efficient, and geometry-aware predictive model remains an open problem, crucial for applications like autonomous navigation and scene understanding.

The paper introduces three key innovations: 1) a z prediction parameterization that directly forecasts clean high-dimensional geometry states, improving stability over velocity prediction; 2) a dual-stage flow curriculum that gradually exposes the model to its own prediction errors, reducing bias accumulation; 3) chunk-wise autoregressive prediction that enables long-horizon forecasting without increasing memory complexity. These innovations leverage the pre-trained VGGT features, which encode multi-view geometry and metric information, making the model inherently geometry-aware. The approach combines continuous flow matching with a novel training curriculum, providing theoretical guarantees and practical robustness in high-dimensional spaces.

• �� Extract deterministic geometry states from observed frames using a frozen VGGT encoder. • Use layer 4 outputs as the predictive state, partitioned into a geometry encoder and feature propagator. • Divide the future prediction into chunks, each predicted autoregressively with a sliding window approach. • Model state transitions via continuous flow matching, predicting the vector field in high-dimensional space. • Employ z prediction to directly forecast clean latent states, avoiding velocity estimation issues. • Implement a dual-stream causal processor and a spatial denoiser to separate temporal reasoning from spatial refinement. • Use a two-stage curriculum: first training with oracle states, then gradually introducing model predictions with partial denoising, to improve long-term stability. • Decode the full predicted trajectory into depth, point clouds, and camera poses, enhancing geometric consistency.

• �� Conduct experiments on KITTI, Cityscapes, and TartanAir datasets, covering depth, point cloud, and camera trajectory tasks. • Baselines include Cosmos, Aether, DINO-Foresight, and geometry-aware models. • Use metrics such as AbsRel and δ1 for depth, Chamfer Distance for point clouds. • Train with AdamW, 0.43B parameters, on a single RTX 6000 GPU, with curriculum-based training. • Ablation studies confirm the importance of z prediction and curriculum design. • Results show significant improvements in accuracy and speed, validating the approach’s effectiveness across diverse scenarios.

• �� Achieved 21% reduction in AbsRel and 5% increase in δ1 over baselines on KITTI, with depth accuracy surpassing 94%. • Model runs 3.6-5× faster than Cosmos with fewer parameters, demonstrating high efficiency. • Ablation confirms that z prediction and curriculum training are critical for stability and performance, especially in high-dimensional space. • Results on Cityscapes and TartanAir further validate robustness and generalization across urban and synthetic scenes.

• �� Enables real-time 3D scene forecasting for autonomous vehicles, robots, and AR/VR systems. • Suitable for scenarios requiring geometric consistency over long horizons without heavy computational costs. • Can be integrated into planning and navigation pipelines, providing foresight in dynamic environments. Future work includes multi-modal fusion and adaptation to more complex, real-world scenarios.

• �� Relies heavily on the quality of pre-trained VGGT features; scene variations or poor pretraining may limit accuracy. • High-dimensional feature space remains sensitive to noise, especially in cluttered or highly dynamic scenes. • Curriculum parameter tuning is delicate, requiring careful calibration for stability. Further research is needed to improve robustness and scalability in diverse applications.