ConceptFusion: Open-set Multimodal 3D Mapping
ConceptFusion leverages foundation models for open-set, multimodal 3D mapping, enabling zero-shot queries with over 40% improvement in 3D IoU.
Key Findings
Methodology
The approach integrates traditional SLAM-based 3D reconstruction with pixel-aligned features extracted from pre-trained foundation models such as CLIP and AudioCLIP. By generating region proposals via models like Mask2Former and SAM, the system computes global and local features, then fuses them using cosine similarity-based weighting to produce pixel-aligned features. These features are integrated into the 3D map through volumetric fusion, enabling real-time, zero-shot, multi-modal querying (text, image, audio, click). The system maintains long-tail concepts better than supervised methods, outperforming them by over 40% in 3D IoU across multiple datasets.
Key Results
- On the UnCoCo dataset, ConceptFusion achieves a 3D IoU of 0.446 for text queries, surpassing LSeg-3D (0.128) and OpenSeg-3D (0.289). In structured indoor scenes, detection accuracy exceeds 70%, with significant improvements in recognizing fine-grained and rare concepts. The system supports multi-modal queries with high precision, demonstrating robust spatial reasoning capabilities in real-world robotic and autonomous driving tasks, with performance gains consistent across diverse environments.
Significance
This work addresses the fundamental limitation of closed-set, supervised 3D mapping systems by enabling open-set, multi-modal reasoning without retraining. Leveraging foundation models' extensive knowledge bases, it significantly enhances scene understanding, object recognition, and spatial reasoning in complex environments. The approach paves the way for more intelligent, adaptable robots and autonomous vehicles capable of nuanced perception and interaction, reducing reliance on extensive labeled datasets and enabling zero-shot generalization.
Technical Contribution
The paper introduces a pixel-aligned feature extraction mechanism that operates in a zero-shot manner, integrating foundation model features into 3D maps via volumetric fusion. It innovates by combining global and region-specific features through a similarity-weighted fusion process, supporting multi-modal queries and spatial reasoning modules like 3DSC. The system achieves real-time performance by optimizing feature extraction and fusion, enabling scalable, open-set scene understanding without additional training or fine-tuning, representing a significant advancement over existing supervised and finetuned methods.
Novelty
This is the first work to realize pixel-level, zero-shot fusion of foundation model features into 3D maps for open-set, multi-modal querying. Unlike prior methods relying on finetuning or region-specific training, it leverages off-the-shelf models to retain extensive knowledge, enabling robust recognition of long-tail and fine-grained concepts. The integration of multi-modal queries and spatial reasoning modules further distinguishes it, offering a comprehensive solution for open-world scene understanding.
Limitations
- The current system's real-time performance is limited by the computational cost of feature extraction (~10-15 seconds per image), restricting deployment in highly dynamic environments.
- Handling rapidly changing scenes or moving objects remains challenging due to reliance on static reconstruction and pre-trained features.
- Accuracy diminishes in scenarios with severe occlusion or extreme clutter, requiring further robustness improvements.
Future Work
Future directions include optimizing feature extraction for faster inference, integrating dynamic SLAM for real-time scene updates, and exploring end-to-end training strategies to improve robustness. Extending the approach to larger-scale, multi-scene maps and incorporating more modalities like tactile or thermal data are promising avenues. Additionally, developing adaptive spatial reasoning modules will enhance understanding of complex spatial relations in dynamic environments.
AI Executive Summary
Building intelligent robots and autonomous vehicles capable of understanding complex environments requires scene representations that go beyond traditional closed-set, supervised methods. Existing systems are limited to recognizing predefined categories, which constrains their ability to handle the diversity and unpredictability of real-world scenes. To address this, Krishna Murthy and colleagues introduce ConceptFusion, a novel framework that leverages foundation models like CLIP and AudioCLIP to create open-set, multimodal 3D maps. This approach integrates pixel-aligned features extracted directly from pre-trained models into real-time SLAM-based reconstructions, enabling the system to understand and query scenes using natural language, images, audio, or clicks, without additional training. The core innovation lies in the pixel-level feature extraction mechanism that combines global and region-specific features via cosine similarity-based weighting, preserving long-tail concepts and fine-grained details. Extensive evaluations on datasets such as UnCoCo, ScanNet, and real-world robotic tasks demonstrate that ConceptFusion significantly outperforms traditional supervised methods, achieving a 3D IoU of 0.446 in text-based object localization, a 40% improvement over prior approaches. The system's ability to support multi-modal queries and spatial reasoning modules like 3DSC opens new avenues for scene understanding, robotic manipulation, and autonomous navigation. This work represents a major step toward truly open-world perception systems that can adaptively interpret complex environments using the vast knowledge embedded in foundation models. Looking ahead, future work will focus on enhancing real-time performance, robustness in dynamic scenes, and scaling to larger, more diverse environments, ultimately enabling smarter, more adaptable autonomous agents.
Deep Analysis
Background
近年来，深度学习推动了3D空间感知的快速发展，代表性技术包括PointNet、VoxNet等。传统方法多依赖封闭集标签，限制了场景理解的广度。基础模型如CLIP、DINO的出现，为开放集、多模态感知提供了新可能，但其在3D映射中的应用仍受限于像素级特征缺失。现有研究多采用微调或区域匹配，难以实现零样本、多模态的高效融合。如何结合预训练模型的知识库，构建支持多模态、开放式查询的3D地图，成为研究热点。
Core Problem
核心问题在于如何将预训练基础模型的全局和区域特征无缝融合到3D重建中，实现零样本、多模态查询。传统方法依赖微调，限制了长尾概念的识别和泛化能力。此外，缺乏高效的像素对齐机制，导致细粒度和长尾概念的丢失，影响空间推理的准确性。这些限制阻碍了机器人和自动驾驶系统在复杂环境中的智能感知能力。
Innovation
提出Pixel-Aligned特征无训练提取机制，结合SLAM实现实时多模态3D映射。创新点包括：1）区域提议与像素对齐，提升细粒度识别；2）特征融合机制，结合全局与局部特征，增强长尾概念保持；3）多模态查询支持，涵盖文本、图像、音频和点击，支持零样本空间推理。这些创新突破了微调依赖，极大提升了开放集、多模态场景的理解能力。
Methodology
- �� 利用深度SLAM（如NVIDIA ∇SLAM）实现实时3D重建，融合深度和颜色信息。
- �� 采用预训练基础模型（CLIP、AudioCLIP）提取像素级特征，通过区域提议（Mask2Former、SAM）获得区域掩码。
- �� 计算每个区域的全局特征fG和局部特征fL，利用余弦相似度进行特征加权，生成像素对齐特征fP。
- �� 将特征融合到点云中，支持多模态查询（文本、图像、音频、点击），实现空间关系推理（如距离、位置关系）。
- �� 设计空间关系模块（3DSC），支持复杂空间推理任务。
- �� 通过多模态编码器，将查询转化为向量，计算与地图特征的相似度，支持零样本检索。
Experiments
在UnCoCo、ScanNet、Replica等数据集上进行评估，比较基线（LSeg、OpenSeg）和微调模型。指标包括3D IoU、空间关系推理准确率。采用不同模态（文本、图像、音频）进行查询，验证零样本能力。还在机器人操作和自动驾驶场景中测试，评估实时性能和空间推理效果。超越微调模型40%以上的性能提升，验证了方法的有效性。
Results
ConceptFusion在UnCoCo数据集的文本查询中，3D IoU达0.446，远超传统闭集模型。结构化场景中，检测准确率达70%以上。长尾概念识别方面，保持了显著优势，尤其在细粒度和少见概念上表现优异。多模态查询（音频、图像）也展现出强大能力，支持复杂空间推理。整体结果证明了零样本、多模态融合的有效性和优越性。
Applications
可应用于机器人导航、室内场景理解、自动驾驶环境感知等。支持多模态交互，提升场景理解深度，减少训练成本。未来可扩展到动态环境、多场景大规模地图构建，推动智能系统自主感知和交互能力的提升。
Limitations & Outlook
系统对动态环境适应性有限，特征提取耗时较长，实时性需优化。复杂遮挡和极端场景下，空间推理准确性仍有待提升。未来需结合动态SLAM和端到端训练，增强鲁棒性和效率。
Plain Language Accessible to non-experts
想象你在一个大厨房里准备做饭。这个厨房里有很多不同的食材、厨具和电器。以前的机器人就像只认识几种常用食材，只能回答“这是苹果”或“那是锅”。但现在，ConceptFusion像是给机器人装上了超级大脑，它可以通过观察厨房的每个角落，理解各种食材的细节，比如“这是罐可乐”或者“那是蓝色的锅”。它还能听到厨房里的声音，比如“咔哒”声，或者看到图片，甚至根据你点击某个物品，告诉你它的位置。它不用事先学会所有的食材，只要你告诉它想找什么，它就能帮你找到。这个技术让机器人变得更聪明，可以在复杂的厨房里找到任何东西，甚至理解一些很少见的食材。这就像给机器人装了一个万能的“百科全书”，让它在任何场景都能快速理解和反应。
ELI14 Explained like you're 14
想象你在学校的科学实验室里玩拼图游戏。以前，你只能用图片上的提示来拼图，不能用别的方法。现在，想象你有一个超级助手，它不仅能看图片，还能听声音、读文字，甚至用手指点你想找的拼图块。这个助手不用事先学会所有拼图，只要你告诉它“找蓝色的块”或者“那是一个苹果”，它就能帮你找到对应的拼图。它还能告诉你两个拼图块距离多远，或者哪个在上面，哪个在下面。这个助手就像是把所有的拼图信息都装进了一个大脑里，随时帮你找到任何你想要的部分。它不需要提前准备，只要你告诉它想要什么，它就能马上帮你找到。这个技术让机器人变得像人一样聪明，可以在复杂的环境中找到任何东西，甚至理解一些很少见的物品。
Abstract
Building 3D maps of the environment is central to robot navigation, planning, and interaction with objects in a scene. Most existing approaches that integrate semantic concepts with 3D maps largely remain confined to the closed-set setting: they can only reason about a finite set of concepts, pre-defined at training time. Further, these maps can only be queried using class labels, or in recent work, using text prompts. We address both these issues with ConceptFusion, a scene representation that is (1) fundamentally open-set, enabling reasoning beyond a closed set of concepts and (ii) inherently multimodal, enabling a diverse range of possible queries to the 3D map, from language, to images, to audio, to 3D geometry, all working in concert. ConceptFusion leverages the open-set capabilities of today's foundation models pre-trained on internet-scale data to reason about concepts across modalities such as natural language, images, and audio. We demonstrate that pixel-aligned open-set features can be fused into 3D maps via traditional SLAM and multi-view fusion approaches. This enables effective zero-shot spatial reasoning, not needing any additional training or finetuning, and retains long-tailed concepts better than supervised approaches, outperforming them by more than 40% margin on 3D IoU. We extensively evaluate ConceptFusion on a number of real-world datasets, simulated home environments, a real-world tabletop manipulation task, and an autonomous driving platform. We showcase new avenues for blending foundation models with 3D open-set multimodal mapping. For more information, visit our project page https://concept-fusion.github.io or watch our 5-minute explainer video https://www.youtube.com/watch?v=rkXgws8fiDs