Fus3D: Decoding Consolidated 3D Geometry from Feed-forward Geometry Transformer Latents
Fus3D leverages pretrained multi-view geometry transformer latents to directly regress dense SDF in under 3 seconds, enabling fast, pose-free 3D reconstruction.
Key Findings
Methodology
This paper introduces a novel approach that extracts 3D geometry directly from the intermediate features of a pretrained multi-view geometry transformer (G). By designing a learned volumetric extraction module (E), multi-view features are fused into a structured 3D latent grid via interleaved cross- and self-attention mechanisms. A simple convolutional head (H3D) then decodes this grid into a dense Signed Distance Field (SDF). Additionally, a validity-aware supervision scheme uses SDFs derived from depth maps or 3D assets to handle non-watertight meshes and unobserved regions, ensuring robustness across sparse and dense view settings.
Key Results
- Fus3D achieves complete, well-defined SDF outputs in under three seconds without camera calibration, outperforming TSDF fusion-based methods like VGGT+TSDF, especially in sparse-view scenarios. Quantitative results on DTU and Objaverse datasets show a 29% reduction in Chamfer distance compared to UFORecon and VolRecon. The method maintains high fidelity even with as few as 2 views and scales well with increasing views, demonstrating robustness and efficiency.
- In experiments, Fus3D successfully reconstructs complex geometries with high surface completeness and detail preservation. It effectively fills unobserved regions and avoids error accumulation typical in multi-view fusion pipelines, providing more accurate and complete 3D models.
- The validity-aware supervision enhances the model's ability to handle non-watertight meshes and unobserved areas, leading to more reliable reconstructions in real-world scenarios, including scenes with open boundaries or incomplete meshes.
Significance
This work fundamentally shifts 3D reconstruction paradigms by removing the dependence on post-hoc fusion of per-view predictions. Instead, it exploits the joint geometric prior encoded within the transformer’s features, enabling rapid, complete, and robust dense SDF estimation without explicit camera parameters. Such advancements significantly impact fields like robotics, AR/VR, and scene understanding, especially in scenarios with sparse or uncalibrated views, where traditional methods struggle to produce coherent models.
Technical Contribution
The core innovation lies in directly extracting 3D geometry from the transformer’s intermediate feature space, bypassing the predict-then-fuse pipeline. The learned volumetric extraction module (E) employs attention mechanisms to fuse multi-view features into a structured latent volume, which is then decoded into SDF. The validity-aware supervision scheme further improves robustness against non-watertight meshes and unobserved regions. This end-to-end, pose-free approach simplifies the pipeline and enhances reconstruction quality, representing a new direction in 3D deep learning.
Novelty
This is the first work to utilize the intermediate features of a pretrained multi-view geometry transformer for dense SDF regression directly. Unlike prior methods that rely on explicit camera calibration or post-processing fusion, Fus3D performs end-to-end volumetric extraction, leveraging the joint multi-view prior learned during training. This approach significantly reduces complexity and improves robustness, especially in sparse and uncalibrated scenarios.
Limitations
- The method depends on a fixed volumetric grid, which may limit scalability to very large scenes or highly complex geometries. Adaptive or hierarchical representations could be explored to address this.
- Performance in extremely sparse views (e.g., fewer than 2) remains challenging, as unobserved regions may not be fully reconstructed.
- Training requires large datasets with high-quality depth or mesh supervision, which can be costly and limit generalization to novel scene types.
Future Work
Future directions include extending the framework to dynamic scenes, incorporating temporal information, and reducing supervision requirements via self-supervised learning. Exploring adaptive volumetric representations and multi-modal data fusion (e.g., LiDAR, RGB-D) could further enhance reconstruction fidelity and scalability. Additionally, integrating this approach into real-time systems for robotics and AR applications remains a promising avenue.
AI Executive Summary
Fus3D introduces a groundbreaking method for dense 3D geometry reconstruction that leverages the rich joint prior encoded within the intermediate features of a pretrained multi-view geometry transformer. Unlike traditional pipelines that rely on per-view predictions followed by post-hoc fusion, Fus3D directly extracts a structured 3D latent volume from transformer features through a learned volumetric extraction module. This approach enables the model to produce complete, well-defined Signed Distance Fields (SDFs) in under three seconds, without requiring camera calibration or explicit pose information.
The core innovation is the transformation of 2D multi-view features into a 3D volumetric representation using attention mechanisms, which fuse information across views into a coherent latent grid. A simple convolutional decoder then maps this grid into dense SDF values, capturing scene geometry with high fidelity. To address practical challenges such as non-watertight meshes and unobserved regions, the authors propose a validity-aware supervision scheme that uses SDFs derived from depth maps or 3D assets, improving robustness in real-world scenarios.
Experimental results on datasets like DTU and Objaverse demonstrate that Fus3D outperforms existing methods such as UFORecon and VolRecon, reducing Chamfer distance by approximately 29% and maintaining high surface completeness across varying view counts. Notably, Fus3D operates in a pose-free, end-to-end manner, making it highly suitable for applications where camera calibration is unavailable or unreliable.
This work marks a significant step forward in 3D scene understanding, offering a fast, scalable, and robust solution that leverages the power of pretrained transformers. Its ability to produce accurate reconstructions from sparse views opens new possibilities in robotics, AR/VR, and autonomous navigation. Looking ahead, future research will explore dynamic scene modeling, multi-modal data integration, and further reduction of supervision needs, aiming to bring this technology closer to real-time, large-scale deployment.
Deep Dive
Abstract
We propose a feed-forward method for dense Signed Distance Field (SDF) regression from unstructured image collections in less than three seconds, without camera calibration or post-hoc fusion. Our key insight is that the intermediate feature space of pretrained multi-view feed-forward geometry transformers already encodes a powerful joint world representation; yet, existing pipelines discard it, routing features through per-view prediction heads before assembling 3D geometry post-hoc, which discards valuable completeness information and accumulates inaccuracies. We instead perform 3D extraction directly from geometry transformer features via learned volumetric extraction: voxelized canonical embeddings that progressively absorb multi-view geometry information through interleaved cross- and self-attention into a structured volumetric latent grid. A simple convolutional decoder then maps this grid to a dense SDF. We additionally propose a scalable, validity-aware supervision scheme directly using SDFs derived from depth maps or 3D assets, tackling practical issues like non-watertight meshes. Our approach yields complete and well-defined distance values across sparse- and dense-view settings and demonstrates geometrically plausible completions. Code and further material can be found at https://lorafib.github.io/fus3d.