WOW-Seg: A Word-free Open World Segmentation Model

The evolution of image segmentation has transitioned from traditional edge and region-based methods to deep learning approaches like FCN (Long et al., 2015), DeepLab (Chen et al., 2018), and transformer-based models. While these methods excel within predefined categories, they struggle with open-world scenarios where object diversity exceeds training data. Recent models like SAM (Kirillov et al., 2023) have introduced class-agnostic segmentation, significantly improving generalization but lacking semantic understanding. Concurrently, vision-language models (e.g., CLIP, Radford et al., 2021) enable zero-shot recognition but are limited by vocabulary scope and multi-instance processing inefficiencies. The gap remains in achieving flexible, semantic-rich, multi-object segmentation without category constraints, motivating the development of models like WOW-Seg that combine visual prompts with large language models for open-world understanding.

The core challenge is enabling accurate, semantic-aware segmentation of arbitrary objects without relying on predefined categories or textual labels. Existing solutions either depend on fixed vocabularies, limiting their scope, or lack the ability to distinguish multiple objects simultaneously, especially in cluttered scenes. Additionally, current datasets do not fully evaluate models' open-world capabilities, often focusing on limited categories. Overcoming these issues requires designing a system that can process multiple instances in parallel, align visual features with semantic space, and generalize to unseen categories, all while maintaining efficiency.

This work introduces three key innovations: 1) Mask2Token converts masks into visual tokens aligned with VLLM features, enabling semantic recognition without text labels. 2) The Cascade Attention Mask mechanism decouples features of multiple instances, reducing interference and supporting parallel inference. 3) Leveraging pre-trained InternVL-1B, the model achieves strong generalization across diverse categories with significantly fewer parameters. These innovations collectively allow the model to recognize any object in open scenes, surpassing existing methods that are limited by predefined categories or inefficient multi-instance handling.

• �� Input: Image and masks (generated by SAM or other methods).
• �� Mask2Token:
• Crop mask regions, resize to 448×448.
• Use shared vision encoder to extract 16×16 feature grid.
• Downsample binary mask to 16×16, guide feature selection.
• Process multiple masks in parallel, map to VLLM feature space.
• �� Cascade Attention Mask:
• Build causal attention with mask-based blocking.
• Ensure each mask’s features are attended only by corresponding tokens.
• During autoregressive decoding, predict each object independently, avoiding cross-instance interference.
• �� Training:
• Fine-tune on large datasets (LVIS, PACO, RR-7K).
• Use self-supervised and supervised signals, optimize for semantic similarity and IoU.
• �� Inference:
• Generate object categories sequentially or in parallel.
• Match predicted features with semantic embeddings for label assignment.

• �� Datasets include LVIS, PACO, and RR-7K, with RR-7K being newly constructed for open-world evaluation.
• �� Training on 8 NVIDIA H100 GPUs, using AdamW optimizer, learning rate 1e-5, batch size 32, for 2 epochs.
• �� Evaluation metrics: semantic similarity, semantic IoU, and parameter efficiency.
• �� Baseline comparisons with SAM, PAM, DAM, showing superior results.
• �� Ablation studies confirm Mask2Token’s and Cascade Attention Mask’s effectiveness.

• �� WOW-Seg achieves 89.7 semantic similarity and 82.4 IoU on LVIS, outperforming previous SOTA by 4.1, with only 1/8 parameters.
• �� On RR-7K, it maintains top performance across 7,662 categories, demonstrating excellent generalization.
• �� The model handles multiple objects simultaneously with reduced interference, validating design choices.
• �� Parameter efficiency and accuracy improvements highlight its practical potential.

• �� Autonomous vehicles benefit from flexible, category-agnostic scene understanding, improving safety and robustness.
• �� Robotics can leverage real-time multi-object recognition for navigation and manipulation.
• �� Medical imaging can utilize open-world segmentation for detecting unknown anomalies.
• �� Long-term, the model could enable intelligent systems capable of understanding diverse environments without retraining.

• �� Performance degrades under severe occlusion or clutter, due to mask generation and feature alignment challenges.
• �� Heavy reliance on large pre-trained models increases computational costs, limiting edge deployment.
• �� Validation is mainly on static images; real-time video and dynamic scene adaptation need further development.