ModuSeg: Decoupling Object Discovery and Semantic Retrieval for Training-Free Weakly Supervised Segmentation
ModuSeg decouples object discovery and semantic retrieval, enabling training-free weakly supervised segmentation with state-of-the-art performance.
Key Findings
Methodology
ModuSeg employs a modular approach, separating object discovery via a geometric mask proposer from semantic assignment through offline feature retrieval. It integrates semantic boundary purification and soft-masked feature aggregation to enhance boundary accuracy and feature purity. Using pre-trained foundation models, it constructs an offline feature bank, avoiding multi-stage training. During inference, class-agnostic proposals are matched with the feature bank via retrieval, assigning semantics without any parameter tuning, thus simplifying the pipeline and reducing computational costs.
Key Results
- On VOC and COCO datasets, ModuSeg achieves 86.6% and 56.7% mIoU respectively, surpassing many training-based methods by 6-7 percentage points, while requiring no training. The method preserves fine boundaries and object details, demonstrating robustness across diverse scenarios. Ablation studies confirm the effectiveness of boundary purification and soft masking, with significant performance gains.
- Generalization tests on ADE20K and Cityscapes show strong transferability. The purified feature bank improves prototype quality, boosting retrieval accuracy. The approach maintains high boundary fidelity and detailed segmentation, validating the modular design’s effectiveness.
- Ablation results indicate that boundary purification and soft-masked aggregation each contribute notably (~3-4%) to overall performance, with their combination yielding optimal results. The retrieval-based paradigm effectively mitigates pseudo-label noise, leading to superior segmentation quality.
Significance
This work advances weakly supervised segmentation by fully leveraging foundation models without training, addressing the bottleneck of pseudo-label noise and boundary ambiguity. The decoupled architecture simplifies the pipeline, reduces training costs, and enhances boundary precision, making it suitable for large-scale applications and real-world deployment. It opens new avenues for training-free, modular segmentation frameworks, fostering broader adoption in industry and research. The approach also demonstrates how pre-trained models can be integrated into downstream tasks, pushing the boundary of zero-shot and few-shot learning in vision tasks.
Technical Contribution
The paper introduces a novel decoupled architecture that separates object localization from semantic classification, utilizing geometric proposals and offline feature retrieval. It innovates with semantic boundary purification and soft-masked feature aggregation, improving feature quality and boundary sharpness. The approach leverages foundation models like CLIP and SAM, constructing a high-quality, discriminative feature bank without any training. During inference, it combines class-agnostic proposals with retrieval-based semantic assignment, enabling pixel-level segmentation with minimal supervision and no additional training, thus broadening the scope of training-free vision methods.
Novelty
This is the first framework to fully decouple object discovery from semantic retrieval in a training-free manner, utilizing foundation models for both geometric proposal and semantic description. The integration of boundary purification and soft masking in a retrieval-based pipeline is novel, effectively addressing pseudo-label noise and boundary ambiguity. Unlike prior methods relying on end-to-end training or multi-stage refinement, ModuSeg’s modular design offers a flexible, scalable solution that leverages pre-trained models’ capabilities directly, representing a significant shift in weakly supervised segmentation paradigms.
Limitations
- The quality of geometric proposals heavily influences performance; inaccurate proposals can degrade segmentation quality. The reliance on pre-trained models may limit adaptability to highly specialized or novel domains. Boundary purification may sometimes remove subtle edge details, affecting fine-grained segmentation. Computational costs of large-scale retrieval and feature bank management could be high for very large datasets. Future work should focus on improving proposal robustness and adaptive feature purification to address these issues.
Future Work
Future directions include enhancing the robustness of geometric proposal methods, integrating multi-modal cues for better object discovery, and refining feature purification to balance boundary detail preservation with noise suppression. Exploring lightweight, adaptive retrieval mechanisms and extending the framework to open-vocabulary scenarios could further broaden applicability. Additionally, combining minimal fine-tuning with the current modular approach might improve performance in highly complex environments, pushing the boundaries of training-free segmentation further.
AI Executive Summary
ModuSeg introduces a groundbreaking approach to weakly supervised semantic segmentation by decoupling object discovery from semantic retrieval, leveraging foundation models in a training-free framework. Traditional methods often rely on complex, multi-stage training pipelines that are sensitive to pseudo-label noise and boundary inaccuracies. In contrast, ModuSeg employs a modular architecture: a geometric mask proposer generates candidate object regions with reliable boundaries, while a pre-trained foundation model constructs an offline feature bank representing semantic categories.
During training, the method applies semantic boundary purification and soft-masked feature aggregation to refine feature prototypes, ensuring high purity and boundary accuracy. These prototypes serve as references during inference, where class-agnostic proposals are matched via retrieval, and semantics are assigned through a voting mechanism. This process effectively separates localization from classification, reducing the impact of pseudo-label noise and eliminating the need for parameter tuning.
Experimental results on VOC and COCO datasets demonstrate that ModuSeg surpasses many state-of-the-art training-based methods, achieving 86.6% and 56.7% mIoU respectively. Its ability to preserve fine details and boundaries without training makes it highly scalable and adaptable. The approach’s modularity and reliance on foundation models open new avenues for scalable, training-free segmentation solutions, with promising applications in autonomous driving, medical imaging, and large-scale scene understanding.
While the framework shows impressive performance, it depends on the quality of geometric proposals and the robustness of feature purification. Future work will focus on improving proposal accuracy, extending to open-vocabulary settings, and reducing computational costs. Overall, ModuSeg marks a significant step toward practical, efficient, and versatile weakly supervised segmentation, leveraging the power of foundation models in a novel, decoupled architecture.
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房准备一道菜，你需要找到所有的食材，然后知道每个食材的名字。传统的方法就像是用一把大锤子，把所有东西都砸在一起，希望能找到你需要的食材，但这样很可能只找到一部分，或者把一些不该用的也弄进去。ModuSeg的方法更聪明，它先用一个专门的工具（几何提议器）找到每个食材的大致轮廓，就像用放大镜仔细观察。接着，它用一个超级厉害的厨师（基础模型）在一个巨大的食材库里检索每个食材的详细信息，就像在大仓库里快速找到你要的调料。为了确保只用到真正的食材，它还会筛掉模糊不清或者不纯净的部分，就像用筛子过滤掉杂质。最后，你可以用这些准确的食材信息，快速做出一道美味的菜肴，而不用反复试错。这种方法既快又准，还能应对各种复杂的厨房环境。
ELI14 Explained like you're 14
想象你在学校的操场上玩捉迷藏，你要找到所有藏起来的小伙伴。以前的方法就像用一张大网，把整个操场罩住，然后一个个去找藏起来的小伙伴，但这个网很笨重，有时候会漏掉一些藏得很好的朋友。ModuSeg的方法就像是用一个特别聪明的望远镜（几何提议器）先找到每个小伙伴的大致位置，然后用一个超级厉害的搜索引擎（基础模型）在一个巨大的相册里快速查找每个朋友的照片。这样一来，你不用每次都重新找，只要看一眼就知道谁藏在哪里。还会用一些过滤和筛选，把模糊不清的地方去掉，确保只找到真正的小伙伴。这样你就可以更快、更准地找到所有藏起来的朋友，玩得更开心！这种方法比以前的方法简单多了，也更聪明！
Abstract
Weakly supervised semantic segmentation aims to achieve pixel-level predictions using image-level labels. Existing methods typically entangle semantic recognition and object localization, which often leads models to focus exclusively on sparse discriminative regions. Although foundation models show immense potential, many approaches still follow the tightly coupled optimization paradigm, struggling to effectively alleviate pseudo-label noise and often relying on time-consuming multi-stage retraining or unstable end-to-end joint optimization. To address the above challenges, we present ModuSeg, a training-free weakly supervised semantic segmentation framework centered on explicitly decoupling object discovery and semantic assignment. Specifically, we integrate a general mask proposer to extract geometric proposals with reliable boundaries, while leveraging semantic foundation models to construct an offline feature bank, transforming segmentation into a non-parametric feature retrieval process. Furthermore, we propose semantic boundary purification and soft-masked feature aggregation strategies to effectively mitigate boundary ambiguity and quantization errors, thereby extracting high-quality category prototypes. Extensive experiments demonstrate that the proposed decoupled architecture better preserves fine boundaries without parameter fine-tuning and achieves highly competitive performance on standard benchmark datasets. Code is available at https://github.com/Autumnair007/ModuSeg.