What a MESS: Multi-Domain Evaluation of Zero-Shot Semantic Segmentation

Semantic segmentation has become a cornerstone task in computer vision, with early methods like FCN, U-Net, and DeepLab setting foundational benchmarks. Recent advances leverage large-scale pretraining, exemplified by models like CLIP, ALIGN, and Florence, which encode visual and textual data into shared spaces, enabling zero-shot learning. Despite these developments, most evaluations remain confined to standard datasets such as ADE20K, Pascal VOC, and Cityscapes, which lack the diversity needed for real-world deployment. The gap between academic benchmarks and practical applications in fields like medicine, remote sensing, and agriculture remains significant, necessitating a broader, more systematic evaluation framework.

Current zero-shot semantic segmentation models struggle to perform reliably across diverse domains and sensor types, especially in specialized fields like medical imaging and earth monitoring. Challenges include poor performance on non-RGB data (thermal, multispectral), difficulty distinguishing semantically similar classes, and high computational costs. These limitations hinder the models' scalability and real-world applicability, emphasizing the need for a comprehensive benchmark that captures the complexity of multi-domain environments. Without such evaluation, progress toward robust, generalizable models remains constrained, risking ineffective deployment in critical applications.

This paper introduces several key innovations: 1) a taxonomy-based dataset classification system that captures task-specific features influencing model performance; 2) the construction of a multi-domain benchmark (MESS) with 22 datasets covering various sensors, scene types, and class vocabularies; 3) a comprehensive evaluation of recent zero-shot models, including CLIP-based architectures like CAT-Seg and SAN, across these datasets; 4) multi-dimensional analysis revealing how factors like sensor modality and semantic similarity affect transferability. These contributions enable a nuanced understanding of model robustness and guide future research toward more adaptable solutions.

• �� Develop a taxonomy by analyzing 120 datasets, focusing on key task characteristics such as sensor type, scene complexity, and class vocabulary. • Select 22 datasets representing diverse applications, ensuring coverage of all taxonomy dimensions. • Implement evaluation protocols using publicly available weights for models like CAT-Seg, SAN, OVSeg, and SAM, without additional training. • Use mean IoU as primary metric, with detailed analysis of performance across domains, sensor types, and class similarities. • Conduct ablation studies to assess the impact of label choices and semantic context. • Analyze inference time and computational costs, comparing single-stage and multi-stage models. • Perform cross-domain performance comparisons to evaluate generalization capabilities.

• �� Datasets span medical (Kvasir-Inst, CHASE DB1), earth monitoring (ISPRS, WorldFloods), agriculture (SUIM, CUB-200), engineering (DeepCrack), and general scenes. • Models evaluated include CLIP-based OVSeg, CAT-Seg, SAN, as well as SAM with grounded bounding boxes. • Metrics focus on mean IoU, inference time, and class discrimination ability. • Experiments conducted on NVIDIA V100S, with no additional fine-tuning, using pre-trained weights. • Performance analyzed across sensor types, class similarity, and scene complexity. • Ablation studies examine label semantics and vocabulary effects on accuracy. • Results highlight strengths and limitations, emphasizing the importance of multi-domain evaluation.

• �� CAT-Seg-L achieves 38.14% average mIoU, outperforming other models in multi-domain settings. • In earth monitoring and medical datasets, scores exceed 48%, nearing supervised performance levels. • RGB images yield the highest accuracy, with multispectral and thermal images showing significant performance gaps. • Models face challenges distinguishing highly similar classes, especially in specialized fields. • Two-stage models like OVSeg are accurate but computationally intensive, limiting real-time use. • Performance varies across datasets, with some models excelling in general scenes but struggling in domain-specific contexts. These findings underscore the importance of diverse evaluation for practical deployment.

• �� Medical diagnostics: aiding radiologists and surgeons with automated organ and lesion segmentation. • Remote sensing: land cover classification, disaster assessment, environmental monitoring. • Agriculture: crop identification, pest detection, yield estimation. • Future integration of multi-modal data (e.g., thermal, electromagnetic) will expand AI’s capabilities in autonomous vehicles, robotics, and surveillance, enabling more reliable and scalable systems across industries.

• �� Limited performance on non-visible spectrum data due to pretraining biases. • Difficulty in differentiating classes with high semantic similarity, affecting fine-grained tasks. • High computational costs for complex models hinder real-time deployment. • Future work should focus on multimodal fusion, model efficiency, and expanding datasets to include more specialized domains, addressing current gaps.