Visual Geometry Foundation-Aware Gaussians for Single-Frame Surround-View Driving Reconstruction

Autonomous driving requires accurate 3D scene reconstruction from limited sensors. Traditional methods used LiDAR and multi-view stereo, but these are costly and sensitive to occlusion. Deep learning approaches like NeRF and Gaussian Splatting have advanced the field, enabling more efficient rendering. However, sparse surround-view setups with minimal overlap pose significant challenges due to weak geometric cues, leading to instability and artifacts. Recent efforts incorporate geometric priors and hybrid representations, but often rely heavily on decoder design, which limits robustness under ego-pose changes. The integration of pretrained foundation models for geometry offers a promising direction, yet remains underexplored in sparse, low-overlap scenarios.

The core issue is how to reliably infer a consistent 3D structure from sparse, low-overlap surround-view images, especially under ego-pose variations. Existing methods struggle with scale drift, view-inconsistency, and rendering artifacts because they depend on weak geometric features and complex decoders. These limitations hinder real-time, high-fidelity scene reconstruction essential for autonomous navigation. The challenge is to develop a stable, scalable approach that leverages prior knowledge and disentangles geometry from appearance, ensuring structural accuracy and visual fidelity in diverse driving conditions.

The paper introduces several key innovations: 1) Incorporating a pretrained VGGT model to extract transferable multi-view geometric tokens, providing strong structural priors. 2) Designing a dual-path neck to separate geometry-supporting features from appearance features, improving robustness in weakly observed regions. 3) Implementing a Scale Warmup strategy to stabilize early geometry learning and prevent scale drift during ego-pose changes. 4) Using a hybrid pixel-volume Gaussian decoder that efficiently produces high-quality, renderable 3D scenes without complex decoder modules. These innovations collectively enhance geometric stability and rendering fidelity in sparse, low-overlap environments.

• �� Extract multi-view geometric tokens from surround images using the pretrained VGGT model, forming the foundation representation. • Use a driving-oriented adapter to predict per-view depth and dense features, back-project depths into 3D points to instantiate the scene structure. • Apply Scale Warmup to stabilize early depth and scale estimates during training. • Employ a dual-path neck to generate separate geometry-consistent and appearance-rich features, which are fused for robustness. • Construct a bounded 3D volume using a mask based on the scene range, focusing on relevant spatial regions. • Use a hybrid Gaussian decoder with pixel and volume heads to generate a complete, renderable scene. • Optimize with multi-level losses including photometric, perceptual, depth, and regularization terms, with warmup supervision in early training. • Render novel views via differentiable projection, ensuring geometric and appearance consistency across viewpoints.

The model was trained and evaluated on the nuScenes dataset, which provides sparse surround-view images with known camera parameters. Metrics included PSNR, SSIM, LPIPS, and PCC for geometric consistency. Baselines included NeRF, 3DGS, and recent feed-forward methods. Ablation studies examined the effects of Scale Warmup, dual-path architecture, and pretrained priors. Hyperparameters were tuned for loss weights and warmup duration. Results showed VGGD outperformed baselines, with notable improvements in visual quality and structural stability, especially under ego-pose shifts. The robustness was validated across diverse driving scenarios, demonstrating generalization.

VGGD achieved a PSNR of 19.8, SSIM of 0.67, LPIPS of 0.43, and PCC of 0.342, surpassing prior methods significantly. The pretrained geometric priors effectively reduced scale drift and view-inconsistency, leading to fewer artifacts and more faithful appearance reconstruction. The dual-path design and scale warmup contributed to enhanced stability, especially in weakly observed regions. Quantitative metrics confirmed the model’s superior performance in both reconstruction quality and geometric consistency, establishing a new state-of-the-art benchmark.

This framework can be directly applied to autonomous vehicle mapping, virtual environment generation, and scene understanding. It requires only a single frame of surround-view images and known camera parameters, making it suitable for real-time deployment. Future integration with multi-frame and multi-modal sensors can further improve accuracy and robustness, enabling more reliable navigation and decision-making in complex environments.

Despite strong results, the approach faces challenges in highly dynamic scenes with rapid motion or severe occlusion, where geometric consistency deteriorates. Dependence on pretrained models may limit adaptability to new environments. Computational costs remain high, affecting real-time applications. Future work should focus on multi-frame fusion, multi-sensor integration, and model optimization for deployment efficiency.