GeoSAM2: Unleashing the Power of SAM2 for 3D Part Segmentation
GeoSAM2 achieves 84.06% mIoU on PartObjaverse-Tiny and 74.42% on PartNetE using multi-view 2D mask prediction for 3D segmentation.
Key Findings
Methodology
GeoSAM2 reframes 3D segmentation as multi-view 2D mask prediction, leveraging normal and point maps with user prompts (clicks/boxes). It uses a SAM2 backbone enhanced with LoRA and residual geometry fusion for cross-view consistency.
Key Results
- GeoSAM2 achieved 84.06% mIoU on PartObjaverse-Tiny, outperforming PartField by ~5%.
- On PartNetE, GeoSAM2 reached 74.42% mIoU, significantly surpassing SAMesh and SAMPart3D.
- Ablation studies show geometry encoding and LoRA optimization contribute ~8% and ~4% mIoU improvement, respectively.
Significance
This method addresses coarse control and high computational cost in existing 3D segmentation, enabling fine-grained, interactive control for applications like robotics and 3D modeling.
Technical Contribution
GeoSAM2 introduces geometry-aware encoding and residual fusion for efficient RGB-to-geometry adaptation, alongside a full-view memory mechanism for cross-view consistency.
Novelty
GeoSAM2 is the first to integrate interactive 2D prompts with 3D segmentation, leveraging LoRA-enhanced geometry encoders for precision and control.
Limitations
- Dependence on normal and point maps may limit performance on textured objects.
- Initial segmentation quality relies on memory initialization, affecting consistency.
- Computational cost remains higher than direct point cloud methods.
Future Work
Future work could explore more efficient geometry encoding, optimize memory mechanisms, and extend to dynamic scene segmentation.
AI Executive Summary
GeoSAM2 is a groundbreaking framework for 3D segmentation, transforming the task into multi-view 2D mask prediction to achieve precision and interactive control.
Existing methods struggle with coarse control and high computational costs. GeoSAM2 addresses these by rendering normal and point maps, processing user prompts (clicks/boxes) with an enhanced SAM2 backbone, and ensuring cross-view consistency through a full-view memory mechanism.
Experimental results show GeoSAM2 achieves 84.06% mIoU on PartObjaverse-Tiny and 74.42% on PartNetE, outperforming state-of-the-art methods. It empowers users with pixel-level control, making it ideal for robotics, 3D modeling, and dynamic scene segmentation.
Deep Analysis
Background
3D segmentation is critical for robotics and modeling but often relies on extensive annotations and lacks fine-grained control. Recent multi-view methods using 2D vision models show promise but face cross-view consistency and controllability challenges.
Core Problem
The core challenge is achieving fine-grained 3D segmentation without full 3D labels while providing interactive control capabilities.
Innovation
GeoSAM2 introduces multi-view 2D mask prediction for 3D segmentation, leveraging geometry-aware encoders and full-view memory mechanisms to address consistency and control issues.
Methodology
- �� Render normal and point maps to capture geometric details
- �� Process user prompts with LoRA-optimized SAM2 backbone
- �� Fuse normal and point-map features using residual connections
- �� Retain all view features in memory for cross-view consistency
- �� Post-process masks with k-NN voting for refinement
Experiments
Experiments use PartObjaverse-Tiny and PartNetE datasets to evaluate performance across diverse categories and dynamic scenarios. Baselines include SAMesh and PartField, with mIoU as the primary metric.
Results
GeoSAM2 achieved 84.06% mIoU on PartObjaverse-Tiny, outperforming PartField by ~5%. On PartNetE, GeoSAM2 reached 74.42% mIoU, significantly surpassing SAMesh and SAMPart3D.
Applications
Applicable to robotics for precise part recognition, 3D modeling for interactive segmentation, and dynamic scene segmentation.
Limitations & Outlook
Dependence on normal and point maps may limit performance on textured objects; initial segmentation quality relies on memory initialization; computational cost remains higher than direct point cloud methods.
Plain Language Accessible to non-experts
Imagine assembling a complex LEGO model. GeoSAM2 acts like a smart assistant, helping you identify specific pieces by analyzing photos from different angles. With simple clicks or boxes, it accurately marks each part, ensuring you never miss a piece.
ELI14 Explained like you're 14
Think of playing a 3D puzzle game! GeoSAM2 is like a super helper that lets you point at a piece, and it instantly highlights it for you. It looks at the puzzle from all angles to make sure every piece is correctly marked. Cool, right?
Glossary
Normal Map
An image representing surface geometry details, capturing object shape.
Used for rendering 3D model features.
Point Map
A 3D coordinate map derived from depth data, representing spatial structure.
Helps ensure cross-view consistency.
LoRA Optimization
A parameter-efficient fine-tuning method using low-rank matrices.
Optimizes geometry-aware encoders.
mIoU
Mean Intersection over Union, measures overlap between predicted and ground-truth masks.
Evaluates segmentation accuracy.
Full-View Memory Mechanism
A strategy retaining all view features for cross-view consistency.
Addresses multi-view segmentation challenges.
Open Questions Unanswered questions from this research
- 1 How to reduce dependence on normal and point maps for textured objects?
- 2 Can this method extend to real-time dynamic scene segmentation?
Applications
Immediate Applications
Robotics
Enables precise part recognition and manipulation, improving task efficiency.
3D Modeling
Supports designers in segmenting and editing specific model regions quickly.
Long-term Vision
Dynamic Scene Segmentation
Extends to real-time segmentation tasks for automated environment understanding.
Abstract
We introduce GeoSAM2, a prompt-controllable framework for 3D part segmentation that casts the task as multi-view 2D mask prediction. Given a textureless object, we render normal and point maps from predefined viewpoints and accept simple 2D prompts - clicks or boxes - to guide part selection. These prompts are processed by a shared SAM2 backbone augmented with LoRA and residual geometry fusion, enabling view-specific reasoning while preserving pretrained priors. The predicted masks are back-projected to the object and aggregated across views. Our method enables fine-grained, part-specific control without requiring text prompts, per-shape optimization, or full 3D labels. In contrast to global clustering or scale-based methods, prompts are explicit, spatially grounded, and interpretable. We achieve state-of-the-art class-agnostic performance on PartObjaverse-Tiny and PartNetE, outperforming both slow optimization-based pipelines and fast but coarse feedforward approaches. Our results highlight a new paradigm: aligning the paradigm of 3D segmentation with SAM2, leveraging interactive 2D inputs to unlock controllability and precision in object-level part understanding.