FlexSplat: Flexible Feed-Forward 3D Gaussian Splatting without Point Cloud Correspondence
FlexSplat introduces a calibration-free 3D Gaussian splatting framework using joint depth and camera prediction, achieving near state-of-the-art results without known poses.
Key Findings
Methodology
FlexSplat employs a jointly trained geometry transformer (VGGT) and Gaussian decoder to predict per-image camera parameters and depth maps. The depth guides the Gaussian parameterization, anchoring the scene representation. Multi-view deformable cross-attention aggregates evidence across views, forming a view-consistent primitive set. The system optimizes depth consistency and residual camera errors via an uncertainty-weighted loss, enabling robust, calibration-free scene reconstruction. The primitive count remains fixed and resolution-independent, decoupled from input size and view number, ensuring efficiency. The approach leverages end-to-end training to adapt geometry and improve robustness against pose and depth estimation errors.
Key Results
- On ShapeNet-SRN and Google Scanned Objects, FlexSplat achieves PSNR within 0.7dB of pose-based methods, matching or surpassing state-of-the-art reconstructions without requiring camera poses or ground-truth depth. It attains the best perceptual quality (LPIPS) on GSO, demonstrating high visual fidelity.
- In multi-view experiments, increasing input views from 1 to 4 improves PSNR by over 5dB, with four-view PSNR reaching 29.73dB, comparable to pose-dependent methods. Ablation studies confirm joint training of geometry transformer and decoder as crucial for performance.
- The primitive budget, set at 10K-20K, remains fixed regardless of input resolution or view count, enabling scalable, compact representations. The model maintains real-time inference speeds (~0.7s for single view, ~1s for four views).
Significance
This work advances uncalibrated 3D scene reconstruction, removing the dependency on camera calibration and ground-truth depth. It offers a scalable, efficient, and robust solution suitable for asset creation, robotics, and AR/VR applications. The method's ability to produce view-consistent, detailed reconstructions from uncalibrated images addresses a longstanding challenge, broadening the applicability of neural scene representations in real-world scenarios. Its compact, resolution-independent Gaussian representation paves the way for scalable 3D modeling in resource-constrained environments.
Technical Contribution
The core innovation lies in integrating a jointly trained geometry transformer with a Gaussian decoder, enabling calibration-free scene reconstruction. The depth-guided Gaussian parameterization stabilizes primitive centers, while multi-view deformable attention ensures view consistency. The fixed, resolution-independent Gaussian budget reduces memory and computational costs. The end-to-end training with depth and rendering losses enhances robustness and generalization, distinguishing this approach from prior methods relying on fixed geometry or known poses.
Novelty
This is the first framework to perform end-to-end, calibration-free multi-view Gaussian scene reconstruction using a jointly trained geometry transformer. Unlike pixel-aligned or point-based methods, FlexSplat leverages a view-agnostic, fixed-size Gaussian set, optimized via multi-view deformable attention. Its ability to predict camera parameters and depth jointly, without prior calibration, marks a significant step forward in uncalibrated 3D modeling, bridging the gap between scene-level pose-free methods and object-centric pose-dependent approaches.
Limitations
- The accuracy heavily depends on the quality of depth prediction; errors in depth can lead to artifacts or incomplete reconstructions, especially in complex scenes with occlusions.
- While the primitive count is fixed and resolution-independent, extremely complex or high-detail scenes may require more primitives, potentially impacting efficiency.
- Training involves joint optimization of multiple components, increasing computational complexity and training time. Real-time inference is achieved but may be limited in highly dynamic or large-scale scenes.
Future Work
Future research could focus on improving depth estimation robustness, especially in occluded or cluttered environments. Extending the framework to scene-level reconstructions and dynamic scenes, integrating temporal consistency, and further optimizing inference speed are promising directions. Additionally, exploring unsupervised or weakly supervised training paradigms could reduce data annotation requirements, broadening real-world applicability.
AI Executive Summary
FlexSplat represents a significant breakthrough in uncalibrated multi-view 3D scene reconstruction. Traditional methods rely heavily on accurate camera calibration and ground-truth depth, which are often unavailable or costly to obtain. In contrast, FlexSplat introduces a novel framework that jointly predicts camera parameters and depth maps directly from uncalibrated images, leveraging a deep geometry transformer (VGGT) and a Gaussian primitive-based scene representation. The core innovation lies in depth-guided Gaussian parameterization combined with multi-view deformable attention, enabling the model to aggregate evidence across views into a view-consistent, compact set of primitives.
This approach is grounded in the insight that a fixed, resolution-independent Gaussian budget can effectively represent complex scenes without growing with input size or view number. The model is trained end-to-end with a combination of reconstruction, depth consistency, and perceptual losses, ensuring robustness against pose and depth estimation errors. Extensive experiments on ShapeNet-SRN and Google Scanned Objects demonstrate that FlexSplat achieves near state-of-the-art performance, matching or exceeding pose-dependent methods in PSNR and perceptual quality, despite operating without any known camera poses.
The results highlight the method’s ability to produce high-fidelity, view-consistent reconstructions from minimal supervision, making it highly suitable for asset creation, robotics, and AR/VR applications where calibration is impractical. The primitive count remains fixed across different resolutions and view counts, ensuring scalability and efficiency. Ablation studies confirm the importance of joint geometry training and depth guidance. While promising, challenges remain in handling highly complex scenes and improving inference speed. Future work aims to enhance depth robustness, extend to scene-level reconstructions, and incorporate temporal dynamics, paving the way for broader deployment of calibration-free 3D scene understanding.
Deep Dive
Abstract
We present FlexSplat, a feed-forward framework for novel view synthesis (NVS) from uncalibrated, object-centric multi-view image collections. A recent line of query-based methods reconstructs a compact set of 3D Gaussians by treating them as transformer queries that are refined with multi-view deformable attention; these methods, however, assume that camera poses are given. FlexSplat removes this assumption: a geometry transformer is trained jointly with the Gaussian decoder to predict per-image camera parameters and depth, which in turn ground a depth-guided Gaussian parameterization and a multi-view deformable cross-attention that aggregates evidence across all input views into a single, view-consistent set of primitives. An uncertainty-weighted depth-consistency objective lets the jointly trained geometry adapt to the reconstruction task, while the cross-view consensus formed during decoding absorbs the residual error of the estimated cameras and depth. The representation uses a compact Gaussian budget that is decoupled from the input resolution - unlike pixel-aligned methods, the primitive count does not grow with the image grid - and is not dictated by the number of views. On ShapeNet-SRN and Google Scanned Objects (GSO), FlexSplat matches or approaches posed state-of-the-art reconstructors while requiring neither camera poses nor ground-truth depth, and matches the best perceptual (LPIPS) quality among the compared methods on GSO. Our results indicate that a jointly trained geometry front-end is sufficient to bring calibration-free operation to query-based Gaussian reconstruction while staying within 0.7 dB PSNR of posed methods and matching their perceptual quality.