CitySeg: A 3D Open Vocabulary Semantic Segmentation Foundation Model in City-scale Scenarios
CitySeg uses local-global cross-attention and hierarchical classification to achieve open-vocabulary semantic segmentation and zero-shot inference for city-scale point clouds.
Key Findings
Methodology
CitySeg employs a local-global cross-attention network to enhance point cloud perception and a hierarchical classification strategy to address annotation inconsistencies. Text modality enables open-vocabulary segmentation.
Key Results
- CitySeg achieves SOTA performance on 9 benchmarks, with mIoU reaching 72.0%, significantly outperforming KPConv and RandLA-Net.
- First model to enable zero-shot inference in city-scale point clouds, segmenting unseen categories without visual data.
- Ablation studies show the local-global cross-attention module improves performance by over 15%.
Significance
This work addresses long-standing challenges of multi-source data distribution and annotation inconsistencies in city-scale point clouds, providing a robust foundation model for UAV perception.
Technical Contribution
Introduces a local-global cross-attention network, hierarchical graph encoder, and two-stage training strategy, significantly improving generalization and fine-grained classification.
Novelty
CitySeg is the first model to align point clouds with text modality for city-scale segmentation, pioneering zero-shot inference without visual data.
Limitations
- The model is sensitive to sparse point clouds, with performance degrading in extremely sparse scenarios.
- Relies on high-quality annotation rules, limiting applicability to unstructured data.
- Training is computationally expensive and time-consuming.
Future Work
Future research could explore more efficient training methods, improve robustness to sparse point clouds, and extend to dynamic real-time segmentation tasks.
AI Executive Summary
CitySeg is an open-vocabulary semantic segmentation foundation model designed for city-scale point clouds, addressing challenges of multi-source data distribution and annotation inconsistencies. By leveraging a local-global cross-attention network, the model integrates local details with global context to enhance perception of sparse point clouds. Additionally, a hierarchical classification strategy constructs a graph structure based on annotation rules to unify semantic granularity across datasets.
Experimental results demonstrate that CitySeg achieves SOTA performance on 9 benchmarks, with mIoU reaching 72.0%, significantly outperforming existing methods. Furthermore, CitySeg is the first model to enable zero-shot inference in city-scale point clouds, segmenting unseen categories without relying on visual data. This capability makes it highly valuable for UAV perception systems.
However, CitySeg has limitations, including sensitivity to sparse point clouds and high computational costs. Future work could focus on optimizing efficiency and extending its applications to dynamic scenarios, unlocking further potential in real-world deployments.
Deep Analysis
Background
Point cloud segmentation has become critical in autonomous driving and UAV perception. Existing methods often rely on visual data and struggle with multi-source generalization. As city-scale scenarios grow in complexity, developing robust models for large-scale point clouds has become a pressing need.
Core Problem
City-scale point clouds face two major challenges: multi-source data distribution disparities hinder generalization, and inconsistent annotation rules complicate semantic alignment. These issues limit cross-domain applicability of existing methods.
Innovation
CitySeg introduces a local-global cross-attention network, hierarchical classification strategy, and two-stage training method, enabling point-text alignment and addressing limitations of prior approaches.
Methodology
- �� Data preprocessing: Standardizes multi-source point cloud density and fills missing attributes.
- �� Local-global cross-attention: Combines local details with global context for enhanced perception.
- �� Hierarchical classification: Constructs a graph structure to unify semantic granularity across datasets.
- �� Two-stage training: Coarse-grained learning followed by fine-grained subclass refinement.
Experiments
Experiments use 9 city-scale datasets, including SensatUrban and STPLS3D. The model is compared against KPConv, RandLA-Net, and others, with ablation studies validating module contributions.
Results
CitySeg achieves 72.0% mIoU on SensatUrban, outperforming KPConv's 57.6%. Ablation studies show the local-global cross-attention module boosts performance by 15%.
Applications
CitySeg is applicable to UAV urban perception, smart city planning, and disaster assessment, especially in scenarios lacking visual data.
Limitations & Outlook
The model is sensitive to sparse point clouds, has high training costs, and relies heavily on annotation rules. Future work could optimize efficiency and extend to dynamic scenarios.
Plain Language Accessible to non-experts
Imagine building a massive LEGO city. Each LEGO piece represents a point in a point cloud, but they come from different sets, with mismatched colors and shapes. CitySeg acts like a master builder, sorting the pieces (local-global cross-attention) and following a detailed guide (hierarchical classification) to assemble a complete city model.
ELI14 Explained like you're 14
Think of Minecraft! You're building a huge city, but your blocks come from different texture packs, with mismatched colors and shapes. CitySeg is like a super-smart AI helper that organizes your blocks and even recognizes new textures you've never used before, helping you build the perfect city effortlessly!
Glossary
Point Cloud
A collection of points in 3D space, each with position and attribute data.
Used to represent city-scale 3D environments.
Local-Global Cross-Attention
An attention mechanism combining local details with global context for enhanced perception.
Used in CitySeg's point network.
Hierarchical Classification
A strategy using graph structures to unify semantic granularity across datasets.
Addresses annotation inconsistencies in CitySeg.
Zero-shot Inference
The ability to predict new categories without training data.
CitySeg achieves this for city-scale point clouds.
mIoU
Mean Intersection over Union, a metric for semantic segmentation accuracy.
Used to evaluate CitySeg's performance on benchmarks.
Open Questions Unanswered questions from this research
- 1 How can robustness to extremely sparse point clouds be improved?
- 2 Can the model be extended to real-time segmentation in dynamic scenarios?
- 3 How can reliance on annotation rules be reduced?
Applications
Immediate Applications
UAV Urban Perception
Enables UAVs to navigate and identify targets in complex urban environments, improving task efficiency.
Smart City Planning
Supports 3D modeling and infrastructure management for optimized urban planning.
Long-term Vision
Real-time Dynamic Segmentation
Future potential to extend to dynamic scenarios, enabling real-time point cloud analysis.
Abstract
Semantic segmentation of city-scale point clouds is a critical technology for Unmanned Aerial Vehicle (UAV) perception systems, enabling the classification of 3D points without relying on any visual information to achieve comprehensive 3D understanding. However, existing models are frequently constrained by the limited scale of 3D data and the domain gap between datasets, which lead to reduced generalization capability. To address these challenges, we propose CitySeg, a foundation model for city-scale point cloud semantic segmentation that incorporates text modality to achieve open vocabulary segmentation and zero-shot inference. Specifically, in order to mitigate the issue of non-uniform data distribution across multiple domains, we customize the data preprocessing rules, and propose a local-global cross-attention network to enhance the perception capabilities of point networks in UAV scenarios. To resolve semantic label discrepancies across datasets, we introduce a hierarchical classification strategy. A hierarchical graph established according to the data annotation rules consolidates the data labels, and the graph encoder is used to model the hierarchical relationships between categories. In addition, we propose a two-stage training strategy and employ hinge loss to increase the feature separability of subcategories. Experimental results demonstrate that the proposed CitySeg achieves state-of-the-art (SOTA) performance on nine closed-set benchmarks, significantly outperforming existing approaches. Moreover, for the first time, CitySeg enables zero-shot generalization in city-scale point cloud scenarios without relying on visual information.