Side Adapter Network for Open-Vocabulary Semantic Segmentation

Semantic segmentation has evolved from early encoder-decoder architectures like U-Net and DeepLab to incorporate large-scale pretraining. Models such as HRNet and SegFormer improved accuracy but still depend on extensive labeled datasets. The advent of vision-language models like CLIP introduced zero-shot recognition, enabling models to classify unseen categories based on text prompts. However, CLIP's image-level training limits pixel-level applications. Two-stage methods attempted to bridge this gap by generating masks first and recognizing classes later, but these approaches are computationally heavy and complex. Recent efforts focus on integrating CLIP features directly into segmentation frameworks, yet challenges remain in balancing model size, inference speed, and recognition accuracy. This context underscores the importance of developing lightweight, end-to-end solutions that can leverage CLIP's zero-shot capabilities for dense prediction tasks.

The core challenge is adapting CLIP's image-level recognition ability to pixel-level semantic segmentation efficiently. Existing methods either require heavy fine-tuning, leading to large models and slow inference, or rely on two-stage pipelines that are cumbersome and less scalable. Achieving high accuracy across a broad vocabulary with minimal parameters remains difficult. Additionally, effectively utilizing deep CLIP features for dense pixel prediction without increasing complexity is an open problem. Addressing these issues is crucial for deploying open-vocabulary segmentation in real-world scenarios where computational resources are limited.

The paper introduces SAN, which innovates by: 1) attaching a lightweight side network to a frozen CLIP backbone, enabling end-to-end training; 2) employing deep feature fusion from shallow CLIP layers to enhance pixel-level features; 3) designing attention bias mechanisms to guide CLIP's self-attention layers for class recognition. These innovations collectively reduce model size, accelerate inference, and improve recognition accuracy, especially for unseen categories. Unlike prior two-stage approaches, SAN's unified framework allows CLIP features to be directly optimized for dense prediction, representing a significant step forward in lightweight open-vocabulary segmentation.

• �� Input images are partitioned into patches and projected into visual tokens. • A lightweight side network with 8 transformer layers predicts mask proposals and generates attention biases. • Mask proposals are obtained via inner product of query and visual features, refined through MLPs. • Attention biases are produced similarly, guiding CLIP's self-attention layers during recognition. • Deep features from shallow CLIP layers are fused into SAN to enrich pixel features. • End-to-end training optimizes mask prediction, feature fusion, and attention bias jointly, enabling CLIP-aware dense segmentation.

Experiments utilize datasets like COCO, ADE20K, Pascal VOC, with training on COCO's 118K images. The model employs ViT-B/16 and ViT-L/14 CLIP backbones, trained with AdamW for 60K iterations, with hyperparameters tuned for optimal performance. Evaluation metrics include mIoU, with comparisons against state-of-the-art methods such as MaskCLIP and OvSeg. Ablation studies analyze feature fusion depth, attention bias design, and training strategies. Results show SAN achieves 12.4 mIoU on ADE-847, surpassing competitors with fewer parameters and faster inference, validating the effectiveness of the proposed architecture.

SAN demonstrates superior performance across benchmarks, with 2.3 points higher mIoU on ADE20K-847 and 94.6% on Pascal VOC, while using only 8.4M trainable parameters. Inference speed is increased 19-fold compared to prior methods. Ablation results confirm deep feature fusion and attention bias as critical for accuracy. The lightweight design enables deployment on resource-constrained devices, making it practical for real-world applications. The method's robustness is validated through multiple dataset evaluations and ensemble strategies, confirming its generalization ability.

This approach is suitable for autonomous vehicles, robotic perception, and multimedia content analysis, where rapid, accurate recognition of numerous categories is essential. It can be integrated into real-time systems with limited computational resources, providing scalable scene understanding. Long-term, SAN could facilitate multi-modal AI systems, enabling richer contextual awareness and adaptive recognition in dynamic environments, thus transforming intelligent automation and human-computer interaction.

Despite its efficiency, SAN's performance may degrade in highly cluttered or occluded scenes, especially for rare or unseen categories. The reliance on deep feature fusion requires careful tuning, and high-resolution image processing still faces challenges due to fixed feature fusion strategies. Future work should explore multi-scale and self-supervised learning techniques to enhance robustness and generalization, especially in diverse real-world scenarios.