Search3D: Hierarchical Open-Vocabulary 3D Segmentation
Search3D constructs hierarchical scene representations for open-vocabulary 3D segmentation, achieving 49.5 AP on MultiScan benchmark.
Key Findings
Methodology
Search3D employs a hierarchical scene graph combining Mask3D-based object proposals, geometric over-segmentation, and SigLIP-based semantic embedding. The scene is represented as a tree with nodes for scene, objects, and parts. Features are extracted at each level using multi-view image crops and pixel-aligned features, then embedded into a shared vision-language space. The model supports flexible text queries by computing similarity scores at object and part levels, enabling multi-granularity search. The approach leverages scene-scale annotations from MultiScan and ScanNet++ to build a benchmark for open-vocabulary scene segmentation.
Key Results
- On the MultiScan benchmark, Search3D achieved an AP50 of 49.5 for scene-scale 3D part segmentation, surpassing baselines like OpenMask3D by 13.8 points, demonstrating its superior ability to handle fine-grained, multi-level queries.
- In ScanNet++, the method outperformed existing approaches in object and part segmentation accuracy, validating the effectiveness of hierarchical feature fusion and pixel-aligned semantic features.
- Ablation studies confirmed that hierarchical structure and multi-view pixel features significantly improve semantic understanding, especially in cluttered or occluded scenes.
Significance
This work advances 3D scene understanding by integrating hierarchical scene graphs with open-vocabulary semantics, enabling robots and AR systems to perform detailed scene analysis beyond object detection. It addresses the critical need for flexible, multi-level scene search, facilitating more natural human-robot interactions and scene comprehension in complex environments. The introduction of a scene-scale benchmark promotes standardized evaluation and future research in open-vocabulary 3D understanding.
Technical Contribution
The paper introduces a novel hierarchical framework combining geometric over-segmentation, multi-view semantic feature extraction, and shared vision-language embedding for multi-level scene querying. It innovates by supporting open-vocabulary search at scene, object, and part levels without requiring prior query knowledge, enabling real-time, flexible scene analysis. The approach also provides a new benchmark dataset for scene-scale 3D part segmentation, fostering community progress.
Novelty
This is the first method to unify hierarchical scene graph construction with open-vocabulary semantic features for multi-level 3D scene search. Unlike prior works focusing solely on object or point-level semantics, Search3D supports scene-scale, part-level, and attribute-based queries, representing a significant step forward in flexible scene understanding.
Limitations
- Despite robust performance, the method still faces challenges in scenes with extreme occlusion or geometric ambiguity, which can hinder part-level segmentation accuracy.
- High annotation costs for scene-scale and part-level labels limit scalability; automating annotation remains an open challenge.
- Real-time inference in large-scale scenes needs further optimization to meet practical deployment requirements.
Future Work
Future directions include integrating multi-modal data such as audio or tactile signals, improving annotation efficiency via semi-supervised learning, and optimizing inference speed. Extending the framework to outdoor scenes and dynamic environments will further broaden its applicability, alongside exploring self-supervised pretraining to reduce reliance on extensive labeled data.
AI Executive Summary
Understanding complex 3D scenes with fine-grained detail is a longstanding challenge in robotics and virtual reality. Traditional methods often focus on recognizing predefined object categories, limiting their adaptability to new environments. Recent advances in vision-language models like CLIP have enabled open-vocabulary recognition at the image level, but extending this to detailed 3D scene understanding remains difficult.
Search3D addresses this gap by introducing a hierarchical scene representation that captures objects, their parts, and attributes within a tree-structured graph. By combining geometric over-segmentation, multi-view semantic feature extraction, and shared vision-language embeddings, the method supports flexible, multi-level queries such as "wooden table" or "chair backrest". This enables a robot to locate specific scene elements based solely on natural language, greatly enhancing scene interaction capabilities.
Experimental results on the MultiScan and ScanNet++ datasets demonstrate that Search3D achieves a 49.5 AP in scene-scale open-vocabulary 3D part segmentation, outperforming existing baselines by a significant margin. The approach's ability to perform detailed, multi-level searches in complex indoor environments marks a substantial step forward in 3D scene understanding. Its hierarchical structure allows for precise retrieval of objects, parts, and attributes, facilitating applications in robotics, AR, and virtual content creation.
This work's broader impact lies in its potential to enable more intuitive human-robot communication and scene manipulation. By supporting open-ended queries at multiple levels of scene detail, it paves the way for smarter, more adaptable AI systems. Future work will focus on scaling to outdoor scenes, reducing annotation costs, and optimizing inference speed, aiming to make this technology practical for real-world deployment.
Deep Analysis
Background
3D场景理解经历了从几何特征提取到语义理解的演变。早期方法如PointNet、PointNet++实现了点云的分类和分割，但受限于类别预定义。近年来，OpenMask3D、Open3DIS等引入实例级分割，提升了对象识别能力。随着视觉-语言模型（如CLIP）崛起，研究开始融合多模态信息，实现跨模态的开放词汇理解。然而，现有方法多集中于对象或点级别，缺乏场景尺度的层级结构和细粒度部件识别，限制了复杂环境的理解能力。
Core Problem
核心问题在于如何在复杂场景中实现多层次、多粒度的开放词汇分割。现有方法多局限于对象或点级别，难以覆盖场景中细粒度部件和属性的识别需求。尤其在机器人交互中，识别对象的不同部分、材质、属性等信息至关重要，但缺乏有效的层级结构和多尺度特征融合机制，导致理解不够细腻，应用受限。
Innovation
本研究提出基于树状层级场景图的多尺度特征融合框架，支持对象、部件和属性的多层次开放词汇搜索。创新点包括：1）构建场景-对象-部件的层级结构，2）引入多视角像素对齐的语义特征，3）结合场景尺度的标注数据，提升细粒度识别能力。这些创新解决了单一尺度模型的局限，增强了模型的表达能力和泛化性。
Methodology
- �� 输入：RGB-D序列和重建几何
- �� 使用Mask3D提取无类别对象实例
- �� 通过图割算法对每个对象进行细粒度分割
- �� 利用SigLIP和Semantic-SAM提取像素对齐的语义特征
- �� 构建树状场景图，节点包括场景、对象和部件
- �� 计算每个节点的开放词汇特征，支持任意文本查询
- �� 层级匹配机制：结合对象和部件特征与文本嵌入进行相似度计算
- �� 通过特征融合和语义合并优化分割结果
Experiments
采用MultiScan和ScanNet++数据集，设计场景尺度和细粒度部件标注，比较Search3D与OpenMask3D、OpenScene等基线。指标包括AP、召回率和匹配准确率，进行消融分析验证特征融合和像素对齐的重要性。模型参数调优基于验证集，确保性能最优。
Results
模型在MultiScan上实现49.5的AP，优于对比方法13.8点，验证其在场景尺度和细粒度分割中的优势。ScanNet++的标注显示，Search3D在对象和部件匹配上显著优于基线，验证了多层次特征融合的有效性。消融实验确认，层级结构和像素对齐特征是性能提升的关键因素。
Applications
该方法适用于机器人自主导航、场景交互、虚拟现实内容生成等场景，能实现复杂环境中的多层次语义理解。只需提供场景的RGB-D数据和文本描述，即可实现高精度、多尺度的场景搜索与识别，为智能系统提供强大支持。
Limitations & Outlook
当前模型在极端复杂或遮挡严重的场景中仍存在识别不足的问题，主要因几何重叠和语义模糊。标注成本较高，扩展到大规模场景存在数据获取难题。实时性方面，推理速度仍需优化以满足实际应用需求。未来需结合自监督学习，提升泛化能力和效率。
Plain Language Accessible to non-experts
想象你在一个大型仓库里整理各种物品。每个物品都可以拆分成不同的部分，比如椅子的腿、靠背、座垫。现在，如果你想找到一把带有木质腿和皮革靠背的椅子，单纯看整体还不够细致。你需要一种方法，既能识别整把椅子，也能找到它的每个部分，还能理解不同材料的区别。Search3D就像是给你装上了一个智能的分类系统，它可以在仓库里快速找到任何你描述的物品或它的某个部分，无论是“有木质腿的椅子”还是“皮革靠背”。它用一种树状结构，把场景分成不同层次，从整体到细节，帮助你更精准地找到想要的东西。这种技术能让机器人更聪明，能理解复杂的场景，甚至可以帮你在虚拟空间中找到各种细节。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的拼图游戏，你要找到特定的拼图片，比如“有蓝色边框的星星”。如果只看整体拼图，很难找到这个细节，但如果你能把拼图分成不同的区域，比如天空、地面、建筑，然后再细分成更小的部分，就容易多了。Search3D就像是这个分层系统，它把3D场景拆成不同层次，从大到小，帮机器人或电脑更聪明地理解每个部分。它可以用一句话描述，比如“木头做的桌子”或“带有金属把手的抽屉”，然后快速找到对应的部分。这就像你用关键词搜索场景中的细节一样，既快又准。这样，机器人就能更好地理解房间、家具，甚至复杂的场景，帮你做事情，比如打扫、搬运或虚拟现实中的互动。
Abstract
Open-vocabulary 3D segmentation enables exploration of 3D spaces using free-form text descriptions. Existing methods for open-vocabulary 3D instance segmentation primarily focus on identifying object-level instances but struggle with finer-grained scene entities such as object parts, or regions described by generic attributes. In this work, we introduce Search3D, an approach to construct hierarchical open-vocabulary 3D scene representations, enabling 3D search at multiple levels of granularity: fine-grained object parts, entire objects, or regions described by attributes like materials. Unlike prior methods, Search3D shifts towards a more flexible open-vocabulary 3D search paradigm, moving beyond explicit object-centric queries. For systematic evaluation, we further contribute a scene-scale open-vocabulary 3D part segmentation benchmark based on MultiScan, along with a set of open-vocabulary fine-grained part annotations on ScanNet++. Search3D outperforms baselines in scene-scale open-vocabulary 3D part segmentation, while maintaining strong performance in segmenting 3D objects and materials. Our project page is http://search3d-segmentation.github.io.