First Shape, Then Meaning: Efficient Geometry and Semantics Learning for Indoor Reconstruction
FSTM employs a two-stage training process with a single SDF to achieve indoor scene geometry and semantic reconstruction, improving speed by 2.3×.
Key Findings
Methodology
FSTM utilizes a unified single SDF model, first performing geometry warm-up using RGB images and geometric cues (normals, depth), then introducing semantic field estimation in a second stage. This decouples geometry and semantics, avoiding the complexity and computational burden of multi-SDF architectures. The geometry warm-up stabilizes the scene reconstruction, while the subsequent joint optimization refines semantic labels. The approach employs a lightweight semantic decoder conditioned on spatial features, with a post-processing step extracting object-level labels via surface mesh analysis. Experiments on Replica and ScanNet++ datasets demonstrate that FSTM outperforms multi-SDF methods in accuracy, robustness, and training speed, achieving 2.3× faster training on Replica and better object recall in complex scenes.
Key Results
- On the Replica dataset, FSTM reduces training time by 2.3× compared to multi-SDF approaches, with Chamfer distance decreasing by approximately 24%, and improves object recall significantly, capturing more scene details and small objects.
- In ScanNet++, FSTM shows enhanced robustness to real-world noise and occlusions, successfully reconstructing more objects and finer details, demonstrating its practical applicability.
- The two-stage training strategy effectively prevents early semantic supervision from degrading geometric accuracy, leading to more stable convergence and higher-quality reconstructions across diverse indoor scenes.
Significance
This work addresses the scalability and efficiency bottlenecks of multi-SDF scene representations, offering a streamlined, high-fidelity alternative. By integrating semantic understanding into a single SDF framework, it enables more accurate and robust scene reconstruction in cluttered indoor environments. The approach significantly advances the state-of-the-art in neural scene understanding, with broad implications for robotics, AR/VR, and digital content creation, where rapid, detailed 3D scene modeling is crucial.
Technical Contribution
The core innovation lies in decoupling geometry and semantic learning via a two-stage process, leveraging geometric priors for initial scene stabilization, then refining semantics jointly. This avoids the computational overhead and gradient discontinuities inherent in multi-SDF architectures. Additionally, the lightweight semantic decoder and post-processing enable dense object-level labeling without complex multi-object coordination. The method demonstrates that a simple, unified SDF model can outperform complex multi-SDF systems in both accuracy and efficiency, providing a scalable solution for large-scale scene reconstruction.
Novelty
This is the first work to demonstrate that a single SDF, combined with a two-stage training process, can achieve object-aware scene reconstruction with high fidelity and efficiency. Unlike prior multi-SDF or Gaussian splatting methods, which face scalability and detail preservation issues, FSTM simplifies the architecture while maintaining or surpassing their performance. Its innovative decoupling of geometry and semantics, along with the post-processing segmentation, sets a new benchmark for efficient, high-quality indoor scene modeling.
Limitations
- The approach relies on pre-trained monocular depth and normal estimators, which may introduce errors in challenging scenes with extreme occlusions or dynamic objects, affecting reconstruction quality.
- While efficient, the method still faces computational challenges in very large-scale environments, requiring further optimization for real-time applications.
- Handling highly dynamic scenes or scenes with significant non-rigid motion remains an open challenge, as the current framework assumes static scenes and relies on static priors.
Future Work
Future directions include integrating multi-modal sensor data (LiDAR, RGB-D) to improve geometric priors, extending the framework to dynamic scenes with temporal consistency, and developing self-supervised training schemes to reduce dependency on pre-trained models. Additionally, optimizing the pipeline for real-time deployment and scaling to larger environments will be key to broader practical adoption.
AI Executive Summary
Neural surface reconstruction has revolutionized 3D scene modeling, with Signed Distance Functions (SDFs) enabling detailed geometric representations. However, existing multi-SDF architectures, while effective at object-level detail, suffer from high computational costs and limited scalability, especially in cluttered indoor environments. To address these challenges, this study introduces FSTM, a novel framework that employs a two-stage training process with a single SDF model to jointly learn geometry and semantics.
The first stage focuses on geometric warm-up, leveraging RGB images and monocular cues such as depth and normals to stabilize scene geometry without semantic interference. This pretraining phase ensures a robust geometric foundation. In the second stage, semantic fields are introduced, conditioned on learned features, and jointly optimized with the geometry. This decoupling prevents early semantic supervision from degrading geometric accuracy, a common issue in joint training. The semantic information is then extracted via a lightweight post-processing step, which assigns object labels directly onto the reconstructed mesh.
Experimental results on the Replica and ScanNet++ datasets demonstrate that FSTM significantly outperforms multi-SDF approaches, achieving 2.3× faster training on Replica and higher object recall in complex scenes. The method shows superior robustness to real-world noise and occlusion, recovering more objects and finer details. Its simplicity, efficiency, and accuracy make it a promising solution for large-scale indoor scene understanding, with applications spanning robotics, AR/VR, and digital content creation. Future work will focus on integrating multi-modal data, handling dynamic scenes, and optimizing for real-time deployment, pushing the boundaries of neural scene reconstruction.
Deep Analysis
Background
近年来，神经场表示技术如NeRF、SDF等在三维场景重建中取得突破，极大提升了重建的细节和效率。早期工作多关注几何表达，后续逐步引入语义信息以实现场景理解。多SDF架构如ObjectSDF++和RICO通过场景分解提高细节，但带来了模型复杂、训练缓慢的问题。Gaussian splatting等新技术在效率方面展现潜力，但在复杂场景中仍存在细节丢失和小物体遗漏的难题。现有方法在大规模、多物体场景中面临计算瓶颈，亟需更简洁高效的解决方案。
Core Problem
多SDF架构在复杂室内场景中存在训练速度慢、模型复杂、细节不足和小物体遗漏的问题。随着场景中物体数量增加，模型的计算成本呈指数级增长，限制了其实际应用。训练过程中多模型优化不稳定，难以保证一致性和鲁棒性。这些问题阻碍了神经场表示在大规模、多物体场景中的推广，亟需一种既高效又能保持高精度的方案。
Innovation
提出单一SDF的两阶段训练策略，先进行几何预热，利用RGB和几何线索稳定几何，再引入语义场进行联合优化，避免多SDF架构的复杂性。几何预热提升几何重建质量，减少训练不稳定性。引入轻量级语义解码器和后处理，实现对象级语义标注，显著增强场景理解能力。整体架构简洁高效，兼顾精度和速度，为神经场表示提供新思路。
Methodology
- �� 预热阶段：利用RGB图像和单目几何线索（深度、法线）优化场景几何，避免早期引入语义干扰。
- �� 联合训练：几何稳定后，加入语义场进行联合优化，利用softmax分类实现对象级语义标注。
- �� 轻量后处理：通过iso-surface提取和射线积分，实现对象级语义分割。
- �� 损失函数：结合Eikonal、平滑正则和深度、法线、语义监督，确保几何和语义的合理性。
- �� 训练策略：采用两阶段固定迭代比例，确保模型收敛稳定，提升重建质量。
Experiments
在Replica（合成）和ScanNet++（实景）两个室内场景数据集上进行评估。比较对象包括MonoSDF、ObjectSDF++、RICO等多SDF方法。指标涵盖Chamfer距离、Hausdorff距离、召回率等。采用不同网络架构（Multi-Res-Grids和深MLP）验证模型的适应性。通过消融实验验证几何预热和两阶段训练的效果，分析训练速度、细节恢复和鲁棒性。
Results
FSTM在Replica数据集上训练速度比多SDF方法快2.3倍，几何重建精度提升显著，Chamfer距离降低约24%。在ScanNet++中，模型成功重建更多小物体，鲁棒性增强，表现出优异的场景理解能力。两阶段训练策略有效缓解早期语义干扰，模型收敛更快，细节更丰富。整体结果表明，单SDF架构结合几何预热在复杂场景中具有明显优势。
Applications
该方法适用于机器人导航、虚拟现实、增强现实、数字资产创建等领域，特别是在需要高精度场景理解和对象识别的应用中。依赖预训练的单目深度和法线信息，适合场景中多视角、多物体的快速重建。未来可结合多模态传感器，拓展到动态场景和大规模环境，推动其在实际中的应用。
Limitations & Outlook
目前依赖预训练模型的几何线索，受限于单目深度和法线估计的准确性。在极端遮挡或动态场景中，几何预热可能不足，导致细节缺失。模型在超大规模场景中的计算成本仍较高，未来需优化算法效率和存储需求，增强泛化能力。
Plain Language Accessible to non-experts
想象你在建造一个复杂的乐高城堡。传统方法就像用很多不同的积木块拼装每个房间和细节，非常繁琐，容易出错。现在，FSTM就像先用一块大块的积木搭出整体轮廓，然后再用小块填充细节。它先快速搭出大体形状，确保结构稳固，再逐步添加颜色和装饰。这种方法既快又稳，不会因为太多细节而变得混乱。最终，你可以用这个大块结构的模型，轻松识别每个房间和装饰，甚至可以在虚拟空间中自由编辑。它让复杂的场景变得简单高效，像搭乐高一样轻松。
ELI14 Explained like you're 14
想象你在用乐高搭一个大城堡。以前的方法就像每个房间都用不同的积木，一个个拼，花费很长时间，还容易出错。现在，科学家们发明了一种新方法，先用一大块积木搭出城堡的轮廓，像画轮廓线一样快，然后再用小块填充细节，比如窗户、门和旗子。这样一来，整个城堡就又快又稳，还能很清楚地看到每个部分。等到完成后，你还能用电脑把城堡变成虚拟的模型，随意观察和修改。这就像用乐高快速搭建，然后用电脑把它变成虚拟城堡一样，既省时间，又能做很多有趣的事情。
Abstract
Neural Surface Reconstruction has become a standard methodology for indoor 3D reconstruction, with Signed Distance Functions (SDFs) proving particularly effective for representing scene geometry. A variety of applications require a detailed understanding of the scene context, driving the need for object-level semantic signals. While recent methods successfully integrate semantic labels, they often inherit the slow training time and limited scalability of multi-SDF learning. In this paper, we introduce FSTM, a unified approach for learning geometry and semantics through a two-step process: a geometry warm-up using RGB inputs and geometric cues, followed by semantic field estimation. By first optimising geometry without semantic supervision, we observe substantial improvements compared to the standard joint optimisation. Rather than relying on specialised modules or complex multi-SDF designs, FSTM shows that a streamlined formulation is sufficient to achieve strong geometric and semantic reconstructions. Experiments on both synthetic and real-world indoor datasets show that our method outperforms multi-SDF approaches. It trains 2.3x faster on Replica, improves robustness to real-world imperfections on ScanNet++, and achieves higher recall by recovering the surfaces of more objects in the scene. The code will be made available at https://remichierchia.github.io/FSTM.