VolFill: Single-View Amodal 3D Scene Reconstruction with Volumetric Flow Matching
VolFill employs a latent diffusion model with TUDF grids to reconstruct complete 3D scenes from a single RGB image, achieving state-of-the-art accuracy.
Key Findings
Methodology
The approach combines a hybrid 3D variational autoencoder (VAE) that compresses sparse TUDF grids into a compact latent space, with a latent diffusion transformer (DiT) trained via flow matching to denoise and generate full scene geometries. It conditions on geometry foundation models, integrating high-level semantic features and explicit visible geometry for robust scene completion. The training employs multi-scale sparse convolutional encoders and decoders, optimizing L1, binary cross-entropy, and KL divergence losses to ensure detailed, consistent reconstructions. The model supports direct surface extraction and occupancy queries at scale, enabling high-fidelity amodal scene completion.
Key Results
- On SCRREAM, VolFill achieves a CD of 2.84, surpassing previous methods like NOVA3R (3.20), with sharper, more coherent scene reconstructions, especially in occluded regions. On NRGB-D, it reduces CD to 4.52, with improved distributional similarity (FPD), demonstrating superior generalization and structural fidelity across diverse indoor scenes.
- Ablation studies confirm that dual conditioning—combining global geometric features and explicit visible geometry—along with the hybrid sparse-dense VAE, are critical for performance gains. The model consistently outperforms baselines in metrics like F-score and coverage, especially in complex occlusion scenarios.
- Qualitative results show cleaner, smoother meshes and more complete point clouds, directly extracted from the structured TUDF grid, avoiding artifacts common in unstructured point cloud methods. The approach demonstrates robustness in large, cluttered scenes with partial observations.
Significance
This work advances the frontier of single-view scene understanding by enabling detailed, physically plausible, and scalable amodal 3D reconstructions. It addresses longstanding limitations of pixel-aligned and point-cloud methods, providing a structured, continuous distance field representation that supports downstream tasks like surface extraction and occupancy queries. Its generative nature opens new possibilities for scene synthesis, virtual environment creation, and robotic navigation, bridging the gap between partial observations and holistic scene comprehension.
Technical Contribution
The paper introduces a novel hybrid 3D VAE architecture that compresses high-resolution TUDF grids into manageable latent vectors, combined with a flow-matching-based latent diffusion transformer conditioned on both global and local geometry. This integration enables stable, high-fidelity scene generation with direct surface extraction. The dual conditioning strategy effectively leverages geometric priors and observed scene cues, improving robustness and generalization. The approach also demonstrates scalable, efficient inference at high resolution, setting new benchmarks in amodal scene reconstruction.
Novelty
This is the first work to integrate latent diffusion models with structured TUDF representations for single-view scene completion. It innovates by combining a hybrid sparse-dense VAE with a flow-matching diffusion process conditioned on geometry foundation models, enabling detailed, scalable, and physically consistent scene reconstructions. Unlike prior point-cloud or layered ray methods, it directly models continuous distance fields, leading to sharper, more complete geometries.
Limitations
- Despite its robustness, the model struggles with extreme occlusions and highly complex topologies, where some hidden structures may be inaccurately inferred. The reliance on large annotated datasets limits its adaptability to novel environments.
- Computational costs remain high, especially during inference at high resolution, which may hinder real-time applications. Further optimization is needed for deployment in resource-constrained scenarios.
- The current framework primarily focuses on indoor scenes; extending to outdoor or dynamic environments presents additional challenges that require future research.
Future Work
Future directions include integrating multi-view inputs for enhanced accuracy, exploring self-supervised learning to reduce annotation dependence, and optimizing inference speed for real-time deployment. Expanding the framework to dynamic scenes and outdoor environments will broaden its applicability, while incorporating multi-modal data (e.g., RGB-D, LiDAR) could further improve robustness and detail in complex scenarios.
AI Executive Summary
Reconstructing complete 3D scenes from a single RGB image remains a fundamental challenge in computer vision, especially when inferring occluded and hidden structures. Traditional pixel-aligned methods are limited to visible surfaces, often producing fragmented or noisy geometries. Recent advances in generative modeling and geometric priors have improved scene understanding, but they struggle with scalability and structural fidelity in complex environments.
This paper introduces VolFill, a novel generative framework that leverages a structured volumetric distance field representation—Truncated Unsigned Distance Function (TUDF)—to achieve holistic scene reconstruction. The core innovation lies in combining a hybrid sparse-dense 3D variational autoencoder (VAE) with a latent diffusion transformer (DiT) trained via flow matching. The VAE compresses high-resolution TUDF grids into a manageable latent space, while the DiT progressively denoises this latent conditioned on both global geometric features from a foundation model (MoGe2) and explicit visible geometry, enabling accurate completion of occluded regions.
Training on large-scale indoor datasets (3D-FRONT, ScanNet++) and evaluated on SCRREAM and NRGB-D benchmarks, VolFill demonstrates superior performance in reconstructing complete scene geometries. Quantitative metrics show a CD of 2.84 on SCRREAM, outperforming previous methods like NOVA3R (3.20). Qualitative results reveal smooth, topologically consistent meshes and detailed point clouds, directly extracted from the structured TUDF grid, avoiding artifacts common in unstructured point-based approaches.
This work significantly advances the field by providing a scalable, high-fidelity, and physically plausible scene understanding framework. Its ability to generate detailed, complete 3D reconstructions from minimal input opens new avenues for robotics, virtual reality, and scene synthesis. Limitations include handling extreme occlusions and computational costs, but future work aims to address these challenges through multi-view integration, efficiency optimization, and extension to outdoor and dynamic scenes.
Deep Dive
Plain Language Accessible to non-experts
想象你在一个工厂里工作，工厂里堆满了各种机器和货物。有时候，你只能看到一部分货物，因为其他的被挡住了。要知道整个仓库里的所有东西，你不能只看一眼，而是要用一些聪明的办法推断出那些被挡住的部分。这个研究就像是让电脑学会用有限的视角，推断出整个仓库的全部布局，包括那些看不见的部分。它用一种特殊的“距离地图”告诉电脑每个位置到最近货物的距离，然后用一种叫“扩散模型”的智能算法，把模糊的线索变得清晰，最终让电脑能完整地“看到”整个仓库的样子。这就像是给电脑装上了“透视眼”，让它能看到平时看不到的东西。这项技术可以帮助机器人更好地理解环境，也能让虚拟现实变得更真实。
ELI14 Explained like you're 14
你知道在游戏里，有时候你只看到一部分地图，但你想知道整个地图的样子吗？这就像是在玩一个迷宫游戏，只能看到前几步，但你希望知道整个迷宫的布局。科学家们也遇到类似的问题：他们用相机拍一张照片，但想知道场景里所有隐藏的部分，比如被遮挡的家具或墙后面。这个研究开发了一种聪明的“推断”方法，让电脑可以根据只看到的部分，猜出隐藏的结构。它用一种叫“距离场”的特殊地图，告诉电脑每个点离表面有多远，然后用一种叫“扩散模型”的技术，把模糊的猜测变得更清楚。这样，电脑就能完整地画出场景的3D模型，包括那些看不见的部分，就像拥有了“透视眼”一样。这个技术可以帮助机器人更好地理解环境，也能让虚拟现实变得更真实。
Abstract
Reconstructing the complete geometry of a scene from a single RGB image remains challenging - especially when inferring hidden structures where visual evidence is incomplete. We introduce VolFill, a generative framework that predicts the 3D structure of the complete scene rather than relying on traditional pixel-aligned regression. Our method utilizes a hybrid 3D VAE to compress sparse truncated unsigned distance function grids into a compact latent space, paired with a latent Diffusion Transformer that denoises this representation to recover the complete scene. We condition the generation on geometry foundation models, leveraging rich spatial priors for robust reasoning. Unlike existing methods limited by per-ray constraints or unstructured point-cloud queries, VolFill provides a structured representation that supports direct surface extraction and occupancy queries at scale. Extensive experiments on the SCRREAM and NRGB-D datasets demonstrate that our approach significantly outperforms current baselines, providing a robust foundation for holistic spatial understanding.