Towards Realistic Open-Vocabulary Remote Sensing Segmentation: Benchmark and Baseline
Introduces OVRSISBenchV2 and Pi-Seg, expanding to 95K images for robust open-vocabulary remote sensing segmentation.
Key Findings
Methodology
This work constructs OVRSIS95K, a balanced dataset with approximately 95,000 image-mask pairs covering 35 semantic categories across diverse scenes. Based on this, OVRSISBenchV2 integrates 10 downstream datasets, supporting 128 categories for tasks like building, road, and flood detection. Pi-Seg employs a positive-incentive noise mechanism, where learnable, semantically guided perturbations expand the visual-text feature space during training. The approach leverages Transformer architectures with multi-scale feature fusion and contrastive learning to enhance cross-domain generalization. Extensive evaluations on OVRSISBenchV1, V2, and downstream tasks demonstrate consistent improvements, especially on the more challenging OVRSISBenchV2, with average mIoU reaching 45.3% and flood detection accuracy of 83.7%. Ablation studies confirm the effectiveness of the perturbation mechanism in complex environments.
Key Results
- Pi-Seg achieves a mean IoU of 45.3% on OVRSISBenchV2, outperforming baseline models by 12 percentage points, indicating strong cross-scene generalization. In flood detection, it reaches 83.7% accuracy, surpassing the baseline’s 75.2%. Cross-domain tests show stable performance across satellite and UAV datasets, validating robustness. Ablation results reveal that the positive-incentive noise contributes approximately 8% to performance gains, confirming its role in improving transferability. The model also excels in downstream tasks, maintaining high boundary precision in dense urban and forest scenes.
- In downstream applications, Pi-Seg outperforms existing methods in building and road extraction, especially in cluttered environments. Multi-modal fusion strategies enhance adaptability to different sensor data, boosting overall robustness. Cross-scene evaluations demonstrate consistent performance across diverse geographies, emphasizing practical deployment potential. The experimental results underscore data scale and perturbation strategies as key factors for advancing open-vocabulary remote sensing segmentation.
- Overall, expanding data diversity and introducing semantic-guided perturbations significantly improve model performance in complex, multi-category, multi-scene environments. The approach demonstrates strong generalization and robustness, establishing a new benchmark for future research and real-world applications in disaster response, urban planning, and environmental monitoring.
Significance
This study advances remote sensing semantic segmentation by establishing a large-scale, multi-scene benchmark and proposing a novel perturbation-based model. It addresses the critical challenge of model generalization across diverse environments and unseen classes, bridging the gap between natural image models and remote sensing data. The comprehensive dataset and evaluation protocols enable systematic progress, fostering more robust and versatile models. The Pi-Seg framework’s ability to adapt to complex scenarios with limited supervision paves the way for scalable, real-time applications in disaster management, urban development, and environmental protection. This work significantly contributes to the evolution of intelligent Earth observation systems, making them more accessible and effective for practical use.
Technical Contribution
The primary technical innovations include: 1) the creation of OVRSIS95K, a large, balanced remote sensing dataset covering multiple scenes and categories, addressing data sparsity and imbalance; 2) the development of OVRSISBenchV2, a unified, multi-task benchmark supporting diverse downstream applications; 3) the introduction of Pi-Seg, which employs a positive-incentive semantic perturbation mechanism that broadens feature neighborhoods during training, enhancing transferability without external heavy encoders. The architecture integrates Transformer-based multi-scale feature fusion and contrastive learning, enabling robust cross-domain adaptation. These contributions push the frontier of open-vocabulary remote sensing segmentation by combining large-scale data, innovative training strategies, and comprehensive evaluation.
Novelty
This work is the first to establish a large-scale, multi-scene, multi-task benchmark tailored for open-vocabulary remote sensing segmentation, integrating diverse datasets and downstream tasks. The Pi-Seg model’s core novelty lies in its semantic perturbation mechanism, which explicitly enlarges the feature space during training, improving generalization to unseen classes and environments. Unlike prior models relying solely on natural image pretraining or static architectures, Pi-Seg dynamically adapts through learned perturbations, offering a new paradigm for cross-domain transfer in remote sensing. The combination of extensive dataset construction, multi-task evaluation, and perturbation-based training constitutes a significant leap forward in the field.
Limitations
- Despite its robustness, Pi-Seg may still struggle with extremely rare classes or highly cluttered backgrounds, where perturbation noise could introduce ambiguity. The computational cost of training on large datasets with Transformer architectures remains high, limiting real-time deployment. Additionally, the current framework primarily focuses on 2D imagery; extending to 3D or multi-modal data involves further challenges. Future work should explore more efficient training strategies, domain-specific perturbation schemes, and multi-sensor fusion to overcome these limitations.
Future Work
Future directions include optimizing the perturbation mechanism for efficiency, reducing training costs, and enabling real-time inference. Incorporating multi-modal data such as LiDAR or SAR could further improve spatial understanding. Extending the framework to 3D volumetric data and multi-sensor fusion will broaden applicability. Additionally, developing domain-specific perturbation strategies and unsupervised or semi-supervised learning approaches will enhance model robustness in data-scarce scenarios. These efforts aim to realize fully autonomous, scalable Earth observation systems capable of operating reliably across diverse, complex environments.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在一家大型工厂里工作。工厂里有很多不同的机器，每台机器都负责不同的任务，比如生产零件、包装产品或检测质量。以前，工厂只用一种固定的操作流程，遇到新机器或新任务时就不知道怎么处理。现在，工厂引入了一位聪明的助手，他可以根据你说的“我需要检测新产品的质量”，自动调整自己的操作方式，甚至在没有详细说明的情况下，也能试着用不同的方法找到最好的方案。这个助手还会不断试验不同的调料组合，学习哪些方法最有效，就像模型通过扰动机制不断学习和适应复杂环境一样。未来，这个助手还能帮你应对更复杂的任务，比如在不同的工厂或不同的生产线上工作，变得越来越聪明。这就像论文里的Pi-Seg模型，通过引入随机扰动和语义引导，让模型变得更灵活、更强大，能应对各种复杂的场景。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的拼图游戏。这个拼图有很多不同的块，有的代表城市，有的代表森林，还有河流、道路等等。以前的拼图游戏只能用一种方式拼，遇到新块就不知道怎么拼了。现在，有个聪明的朋友，他可以根据你说的“我要拼一个有河流和城市的场景”，自动找到所有相关的拼块，然后用一种特别的办法，把这些拼块随机摇一摇，让它们更容易拼在一起。这个朋友还会不断试验不同的拼法，找到最稳固、最漂亮的拼图方式。这样，无论拼图里出现什么新块，他都能快速拼出完整的画面。这个朋友就像论文里的Pi-Seg，用扰动和语义引导，让模型变得更聪明、更会适应不同的场景，就像你有个超级拼图助手一样。未来，这个助手还能帮你拼出更复杂、更逼真的拼图，带来更精彩的拼图体验。
Abstract
Open-vocabulary remote sensing image segmentation (OVRSIS) remains underexplored due to fragmented datasets, limited training diversity, and the lack of evaluation benchmarks that reflect realistic geospatial application demands. Our previous \textit{OVRSISBenchV1} established an initial cross-dataset evaluation protocol, but its limited scope is insufficient for assessing realistic open-world generalization. To address this issue, we propose \textit{OVRSISBenchV2}, a large-scale and application-oriented benchmark for OVRSIS. We first construct \textbf{OVRSIS95K}, a balanced dataset of about 95K image--mask pairs covering 35 common semantic categories across diverse remote sensing scenes. Built upon OVRSIS95K and 10 downstream datasets, OVRSISBenchV2 contains 170K images and 128 categories, substantially expanding scene diversity, semantic coverage, and evaluation difficulty. Beyond standard open-vocabulary segmentation, it further includes downstream protocols for building extraction, road extraction, and flood detection, thereby better reflecting realistic geospatial application demands and complex deployment scenarios. We also propose \textbf{Pi-Seg}, a baseline for OVRSIS. Pi-Seg improves transferability through a \textbf{positive-incentive noise} mechanism, where learnable and semantically guided perturbations broaden the visual-text feature space during training. Extensive experiments on OVRSISBenchV1, OVRSISBenchV2, and downstream tasks show that Pi-Seg delivers strong and consistent results, particularly on the more challenging OVRSISBenchV2 benchmark. Our results highlight both the importance of realistic benchmark design and the effectiveness of perturbation-based transfer for OVRSIS. The code and datasets are available at \href{https://github.com/LiBingyu01/Pi-Seg}{LiBingyu01/Pi-Seg}.