Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation
Mosaic3D introduces a novel pipeline and model, generating 5.6M high-quality 3D mask-text pairs, achieving SOTA open-vocabulary 3D segmentation.
Key Findings
Methodology
Mosaic3D integrates Grounded-SAM and SEEM for precise 3D segmentation and employs Osprey for detailed region-aware captions. By applying this pipeline to datasets like ScanNet and Matterport3D, the team created Mosaic3D-5.6M, a dataset with 5.6M mask-text pairs. The model uses SparseUNet as a 3D encoder, aligning point features with text embeddings via contrastive learning, and incorporates a lightweight mask decoder for open-vocabulary 3D segmentation.
Key Results
- Result 1: Mosaic3D achieved 15.7% f-mIoU on ScanNet200, outperforming RegionPLC by 4.9 percentage points.
- Result 2: On Matterport3D, Mosaic3D reached 13.1% f-mIoU, 6.9 points higher than prior methods.
- Result 3: Ablation studies confirmed dataset scale and quality significantly impact performance, especially with ARKitScenes and ScanNet++.
Significance
Mosaic3D addresses the lack of large-scale, high-quality datasets for open-vocabulary 3D segmentation, enabling significant performance improvements. Its automated pipeline reduces annotation costs, providing critical infrastructure for 3D vision-language research. Furthermore, it pioneers single-stage open-vocabulary 3D instance segmentation without manual labels, opening new research directions.
Technical Contribution
Key contributions include: 1) precise segmentation using Grounded-SAM and SEEM; 2) detailed region-aware captions via Osprey; 3) the creation of Mosaic3D-5.6M, the largest 3D mask-text dataset; 4) a SparseUNet-based language-aligned 3D encoder and lightweight mask decoder for single-stage segmentation.
Novelty
Mosaic3D is the first single-stage open-vocabulary 3D instance segmentation model, eliminating reliance on multi-view 2D inference. Its pipeline uniquely combines state-of-the-art segmentation and vision-language models to achieve unprecedented data quality and scale.
Limitations
- Limitation 1: The dataset is focused on indoor scenes, lacking coverage of outdoor environments.
- Limitation 2: Performance on long-tail categories remains suboptimal, particularly for rare objects.
- Limitation 3: The computational cost of contrastive learning is high, limiting scalability.
Future Work
Future work includes extending the pipeline to outdoor scenes, improving performance on long-tail categories, and exploring more efficient training methods to reduce computational overhead.
AI Executive Summary
Mosaic3D tackles the challenge of open-vocabulary 3D scene understanding by addressing the lack of large-scale, high-quality datasets. The team developed an automated pipeline combining Grounded-SAM and SEEM for precise segmentation and Osprey for detailed region-aware captions, resulting in Mosaic3D-5.6M, a dataset with 5.6M mask-text pairs.
Building on this dataset, the researchers designed the Mosaic3D model, which uses SparseUNet for language-aligned 3D encoding via contrastive learning and a lightweight mask decoder for single-stage open-vocabulary 3D segmentation. Experiments on benchmarks like ScanNet200 and Matterport3D demonstrate state-of-the-art performance, with significant improvements over existing methods.
This work not only advances academic research in 3D vision but also has practical implications for robotics, AR/VR, and autonomous systems. However, challenges remain, including limited outdoor scene coverage and high computational costs, which the authors aim to address in future work.
Deep Analysis
Background
3D scene understanding is crucial for applications like robotics and AR/VR. Traditional methods rely on predefined categories, but open-vocabulary approaches are needed for more flexible applications. However, the lack of large-scale, high-quality 3D datasets has hindered progress compared to 2D vision-language models trained on web-scale image-text pairs.
Core Problem
The core challenge is the scarcity of datasets that meet three criteria: precise 3D region segmentation, rich textual descriptions, and sufficient scale. Existing methods either rely on expensive manual annotations or produce low-quality masks and captions, limiting their utility for training high-performance models.
Innovation
Mosaic3D introduces: 1) a novel pipeline combining Grounded-SAM and SEEM for precise segmentation; 2) region-aware captions generated by Osprey; 3) Mosaic3D-5.6M, a 5.6M mask-text pair dataset; 4) a SparseUNet-based 3D encoder with contrastive learning and a lightweight mask decoder for single-stage segmentation.
Methodology
- �� Data generation: Combines Grounded-SAM and SEEM for segmentation, and Osprey for detailed captions.
- �� Dataset creation: Annotates ScanNet, Matterport3D, and others to produce 5.6M mask-text pairs.
- �� Model design: SparseUNet aligns 3D point features with text embeddings using contrastive learning.
- �� Mask decoder: A lightweight module predicts instances directly from language-aligned features.
Experiments
Experiments used datasets like ScanNet and Matterport3D, evaluating open-vocabulary semantic and instance segmentation. Metrics include f-mIoU and f-mAcc. Ablation studies examined the impact of dataset scale and quality.
Results
Mosaic3D achieved 15.7% f-mIoU on ScanNet200, outperforming RegionPLC by 4.9 points. On Matterport3D, it reached 13.1% f-mIoU, 6.9 points higher than prior methods. Dataset scale and quality were shown to significantly influence performance.
Applications
Mosaic3D can be applied to AR/VR, robotics, and autonomous systems, particularly for tasks requiring high-precision 3D scene understanding like indoor navigation and virtual environment construction.
Limitations & Outlook
The dataset is limited to indoor scenes, lacks outdoor coverage, and struggles with long-tail categories. Additionally, the computational cost of contrastive learning is high, posing scalability challenges.
Plain Language Accessible to non-experts
Imagine organizing a massive warehouse filled with items. Mosaic3D is like a super-smart robot that not only finds each item but also describes its color, shape, and location. For example, it might say, 'Here’s a blue couch with two yellow pillows.' It scans every corner of the warehouse, uses advanced vision-language models to generate detailed descriptions, and stores this information in a huge database. This way, it can quickly find and describe anything you need.
ELI14 Explained like you're 14
Think of playing a 3D video game with lots of rooms and objects. Mosaic3D is like a super helper in the game that finds all the treasures for you! Not only that, it tells you what each treasure looks like, like 'This is a red chest with a golden key on top!' It’s like a detective scanning the whole map and giving you all the details. Cool, right?
Glossary
Grounded-SAM
A segmentation algorithm combining Grounding-DINO and SAM for precise foreground masks.
Used to generate accurate segmentation masks in 3D scenes.
SEEM
An open-vocabulary panoptic segmentation model handling both foreground and background objects.
Complements Grounded-SAM for background segmentation.
Osprey
A region-aware vision-language model that generates detailed region descriptions.
Used to create captions for each segmented region.
SparseUNet
An efficient 3D point cloud encoder based on sparse convolutional networks.
Serves as the 3D encoder in the Mosaic3D model.
Mosaic3D-5.6M
A 3D dataset with 5.6M mask-text pairs, the largest of its kind.
Used for training and evaluating the Mosaic3D model.
Open Questions Unanswered questions from this research
- 1 How can the pipeline be extended to outdoor scenes?
- 2 How can performance on long-tail categories be improved?
- 3 Are there more efficient contrastive learning methods to reduce computational costs?
Applications
Immediate Applications
Indoor Navigation
Helps robots identify and navigate complex indoor environments, improving efficiency.
Virtual Reality
Enables real-time 3D segmentation and object recognition in virtual environments.
Long-term Vision
General 3D Scene Understanding
Develop universal 3D vision systems for autonomous vehicles and smart homes.
Abstract
We tackle open-vocabulary 3D scene understanding by introducing a novel data generation pipeline and training framework. Our method addresses three critical requirements for effective training: precise 3D region segmentation, comprehensive textual descriptions, and sufficient dataset scale. By leveraging state-of-the-art open-vocabulary image segmentation models and region-aware Vision-Language Models, we develop an automatic pipeline that generates high-quality 3D mask-text pairs. Applying this pipeline to multiple 3D scene datasets, we create Mosaic3D-5.6M, a dataset of over 30K annotated scenes with 5.6M mask-text pairs, significantly larger than existing datasets. Building upon this data, we propose Mosaic3D, a foundation model combining a 3D encoder trained with contrastive learning and a lightweight mask decoder for open-vocabulary 3D semantic and instance segmentation. Our approach achieves state-of-the-art results on open-vocabulary 3D semantic and instance segmentation tasks including ScanNet200, Matterport3D, and ScanNet++, with ablation studies validating the effectiveness of our large-scale training data.