Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP
Proposes mask-adapted CLIP with mask prompt tuning, achieving 29.6% mIoU on ADE20K-150, outperforming state-of-the-art by 8.5%.
Key Findings
Methodology
This paper analyzes two-stage open-vocabulary segmentation, identifying CLIP's poor performance on masked images as a bottleneck. It collects diverse mask-category pairs from COCO Captions via weakly supervised matching, then applies mask prompt tuning—replacing zero tokens in masked patches with learnable prompts—without modifying CLIP weights. Full finetuning combined with prompt tuning significantly boosts performance, achieving 29.6% mIoU on ADE20K-150, surpassing previous SOTA by 8.5%. The approach maintains CLIP's generalization, enabling zero-shot transfer across datasets.
Key Results
- On ADE20K-150, the model reaches 29.6% mIoU, an 8.5% improvement over prior methods. On ADE20K-847 and Pascal Context-459, it achieves 9.0% and 12.4% mIoU, surpassing previous bests by 2.7% and 3.4%.
- Using automatically collected noisy data from captions, the model retains strong open-vocabulary capabilities. Mask prompt tuning avoids weight modifications, supporting multi-task learning.
Significance
This work overcomes the domain gap challenge of applying pretrained CLIP to masked regions, enabling robust open-vocabulary segmentation without dataset-specific tuning. It advances generalist models to match supervised counterparts from 2017, with broad implications for scalable scene understanding in real-world applications like autonomous driving and robotics.
Technical Contribution
Introduces mask prompt tuning—learnable prompts replacing zero tokens in masked patches—coupled with weakly supervised mask-category pair collection. This strategy enhances CLIP's adaptation to masked images without weight updates, combining full finetuning and prompt tuning for optimal performance. The method sets a new benchmark for open-vocabulary segmentation, emphasizing efficiency and generalization.
Novelty
First to leverage caption-derived noisy mask-category pairs for training, combined with a novel prompt tuning mechanism that preserves CLIP's generalization. Unlike prior closed-set or purely finetuning approaches, this method maintains zero-shot capabilities while improving masked image classification.
Limitations
- Performance drops in highly occluded or cluttered scenes due to limitations in mask proposal quality and matching accuracy.
- Noisy data collection from captions may introduce label noise, affecting fine-grained recognition.
- Training costs remain high; further efficiency improvements are needed for large-scale deployment.
Future Work
Future directions include refining mask proposal quality, exploring self-supervised data augmentation, and extending to video and 3D scene understanding. Integrating multimodal cues and reducing computational costs are also promising avenues.
AI Executive Summary
This research addresses a key bottleneck in open-vocabulary semantic segmentation: the poor performance of pretrained CLIP on masked images. Traditional two-stage methods generate class-agnostic masks and classify them using CLIP, but the domain gap between natural images and masked proposals limits accuracy. To overcome this, the authors propose a novel approach: collecting diverse mask-category pairs from image captions using weakly supervised matching, then applying mask prompt tuning—replacing zero tokens in masked patches with learnable prompts—without modifying CLIP weights. This method effectively adapts CLIP to masked regions, preserving its generalization ability.
Extensive experiments demonstrate that combining full model finetuning with mask prompt tuning yields a significant performance boost. On ADE20K-150, the model achieves 29.6% mIoU, outperforming previous state-of-the-art by 8.5%. The approach generalizes well across datasets, including ADE20K-847 and Pascal Context-459, setting new benchmarks. Importantly, the method maintains zero-shot transfer capabilities, enabling open-vocabulary segmentation without dataset-specific tuning.
This breakthrough paves the way for more flexible, scalable scene understanding systems capable of recognizing unseen categories in diverse environments. By leveraging weakly supervised caption data and innovative prompt tuning, the authors demonstrate a practical, efficient path toward universal visual perception models. The work holds promise for applications in autonomous vehicles, robotics, and intelligent surveillance, where adaptability and broad recognition are critical. Despite current limitations in occlusion handling and noisy data, future research will focus on refining data collection, enhancing proposal quality, and extending to multimodal and temporal domains, further pushing the boundaries of generalist AI systems.
Deep Analysis
Background
语义分割作为计算机视觉的基础任务，经过深度学习的推动已取得显著进展。早期方法依赖于像素级标注，受限于类别固定，难以泛化到未知类别。预训练模型如CLIP通过大规模图文数据学习多模态特征，推动了开源词汇理解的发展。两阶段方法结合区域提议和预训练模型，成为主流，但在掩码区域表现仍受域差异影响。现有研究多关注闭集类别，缺乏对未知类别的泛化能力，且在掩码处理上存在挑战。本研究旨在突破预训练模型在掩码区域的性能瓶颈，推动通用视觉理解。
Core Problem
核心问题在于CLIP在掩码区域表现不佳，主要因掩码区域的空白和噪声区域与训练图像差异大，导致域适应困难。传统微调依赖闭集标注，限制了模型的开放词汇能力。如何在保持泛化的同时，有效适应掩码区域，成为关键难题。解决方案需利用丰富的图文描述信息，构建多样化训练数据，并设计高效的模型适应策略。这对于实现真正的通用视觉理解具有重要意义。
Innovation
本文提出掩码提示调优（mask prompt tuning），引入可学习的提示嵌入，缓解掩码区域的域差异，避免修改CLIP权重。结合从图像描述中自动采集的噪声丰富的掩码-类别对，增强模型的开放词汇能力。创新点包括：1）利用弱监督匹配策略自动采集多样化样本，突破闭集限制；2）设计掩码提示机制，有效利用空白区域信息；3）结合全模型微调与提示调优，提升性能。这些创新共同推动开源词汇语义分割的发展。
Methodology
- �� 数据采集：从COCO Captions提取名词，利用预训练的MaskFormer生成掩码区域，结合CLIP匹配图像区域与名词，自动构建噪声丰富的掩码-类别对。• 掩码微调：将掩码区域输入CLIP，使用可学习的提示嵌入替代空白区域的零值掩码，避免模型权重修改。• 全模型微调：冻结CLIP文本编码器，在图像编码器和提示参数上微调，提升掩码区域识别能力。• 融合策略：将微调模型与原始MaskFormer结合，通过融合模块实现最终分割。• 训练细节：采用AdamW优化器，结合多尺度数据增强，训练120K轮，确保多场景泛化。
Experiments
在COCO-Stuff和COCO Captions上训练，评估在ADE20K、Pascal VOC和Pascal Context等公开数据集。指标为mIoU，比较不同微调策略。通过消融实验验证噪声数据贡献和提示调优效果。参数设置包括学习率、批次大小、训练轮数等。结果显示，掩码提示调优在保持泛化的同时，显著提升掩码区域分类准确率。
Results
在ADE20K-150上达29.6%的mIoU，超越前沿8.5%；在ADE20K-847和Pascal Context-459分别实现9.0%和12.4%的新高，优于现有方法2.7%和3.4%。引入噪声丰富的图文匹配数据显著改善了模型的开放能力。掩码提示调优无需修改CLIP权重，支持多任务。多场景验证显示模型具有强泛化能力，展现实际应用潜力。
Applications
该方法适用于自动驾驶、智能监控、机器人感知等场景，实现无需大量标注的场景理解。利用图文描述，模型能识别未知类别，提升系统适应性。未来结合视频和3D场景，有望推动全场景通用视觉AI的发展。
Limitations & Outlook
模型在极端遮挡和复杂背景下表现仍有限，主要因掩码生成和匹配的局限。噪声数据虽丰富，但在某些类别上存在偏差。训练成本较高，需优化采集策略和模型效率。
Plain Language Accessible to non-experts
想象你在一个工厂里，工人们每天都在组装不同的产品。有时候，工厂里会出现一些不完整的零件或被遮挡的部分，工人们需要根据描述判断这些零件属于什么。传统的方法就像是只认固定的零件类型，比如只认苹果或橙子，但遇到新零件就不知道了。现在，这个新方法像是给工人们提供了一个智能助手，它可以根据工厂的描述，自动学习识别各种不同的零件，即使是以前没见过的。这个助手通过不断学习不同的零件和描述，变得越来越聪明，甚至能在零件被遮挡或变形时，仍然准确识别。它还用一种特别的“提示”方式，帮助自己更好地理解那些被遮挡或空白的部分。最终，这个工厂的工人们可以更快、更准地完成任务，工厂也变得更智能、更高效。
ELI14 Explained like you're 14
想象你在学校里，有一个超级聪明的朋友，他可以帮你认出任何你看到的东西。比如你看到一只奇怪的动物，他能告诉你那是只“狐狸”或者“猫”。以前，朋友只能认出一些常见的动物，但如果遇到新动物，他就不知道了。现在，这个新方法就像给朋友装上了一个超级大脑，让他可以通过听你描述的内容，学会认出各种新东西。比如你说“那是一只带有橙色毛发的动物”，朋友就会用这个信息，帮你认出那是只“橙色狐狸”。它还学会了在图片被遮挡或部分看不清时，依然能猜出是什么。这样一来，无论是图片里有遮挡，还是描述不完整，朋友都能帮你认出东西。这就像让普通的朋友变成了超级侦探，能认出各种新奇的东西，非常厉害！
Abstract
Open-vocabulary semantic segmentation aims to segment an image into semantic regions according to text descriptions, which may not have been seen during training. Recent two-stage methods first generate class-agnostic mask proposals and then leverage pre-trained vision-language models, e.g., CLIP, to classify masked regions. We identify the performance bottleneck of this paradigm to be the pre-trained CLIP model, since it does not perform well on masked images. To address this, we propose to finetune CLIP on a collection of masked image regions and their corresponding text descriptions. We collect training data by mining an existing image-caption dataset (e.g., COCO Captions), using CLIP to match masked image regions to nouns in the image captions. Compared with the more precise and manually annotated segmentation labels with fixed classes (e.g., COCO-Stuff), we find our noisy but diverse dataset can better retain CLIP's generalization ability. Along with finetuning the entire model, we utilize the "blank" areas in masked images using a method we dub mask prompt tuning. Experiments demonstrate mask prompt tuning brings significant improvement without modifying any weights of CLIP, and it can further improve a fully finetuned model. In particular, when trained on COCO and evaluated on ADE20K-150, our best model achieves 29.6% mIoU, which is +8.5% higher than the previous state-of-the-art. For the first time, open-vocabulary generalist models match the performance of supervised specialist models in 2017 without dataset-specific adaptations.