CoupAlign: Coupling Word-Pixel with Sentence-Mask Alignments for Referring Image Segmentation
CoupAlign couples sentence-mask and word-pixel alignments, improving oIoU by ~2% on RIS datasets.
Key Findings
Methodology
CoupAlign employs a hierarchical cross-modal alignment framework integrating Word-Pixel Alignment (WPA) in intermediate encoder layers for early fusion, generating mask proposals from aligned features. The Sentence-Mask Alignment (SMA) module weights these masks via sentence embeddings, refining object localization. The architecture combines Swin Transformer for visual features and BERT for language, with multi-scale feature fusion and a mask generator built on a transformer decoder. An auxiliary contrastive loss enforces pixel-wise semantic consistency, enhancing mask integrity. This design effectively captures pixel coherence at both visual and semantic levels, leading to more accurate segmentation.
Key Results
- On RefCOCO validation and test sets, CoupAlign achieves approximately 2% higher oIoU than SOTA methods like LAVT, demonstrating superior localization and mask quality. It performs especially well in multi-object scenarios, accurately distinguishing multiple similar objects, such as multiple persons or similar objects in crowded scenes.
- On G-Ref, CoupAlign surpasses LAVT by 1.6% and 0.13% on validation and test sets, respectively, indicating robustness in complex sentence structures and fine-grained pixel alignment. Ablation studies confirm that combining word-pixel and sentence-mask alignments yields significant performance gains, with mask proposals and hierarchical alignment being critical components.
- The experiments validate that the hierarchical approach reduces fragmentation and boundary errors, with the auxiliary contrastive loss further improving pixel-wise semantic consistency, leading to more complete and precise masks.
Significance
This work advances the state-of-the-art in referring image segmentation by integrating multi-level cross-modal alignments, addressing pixel coherence and multi-object discrimination challenges. Its hierarchical design enhances both accuracy and robustness, facilitating applications in image editing, autonomous driving, and human-robot interaction. The method’s ability to distinguish multiple similar objects under complex conditions marks a significant step forward in fine-grained multimodal understanding, bridging the gap between semantic comprehension and pixel-level precision.
Technical Contribution
The core innovation lies in coupling sentence-mask and word-pixel alignments within a multi-scale hierarchical framework, enabled by intermediate-layer cross-attention modules. The mask proposals generated from aligned features serve as object hypotheses, refined through sentence-based weighting. The auxiliary contrastive loss enforces intra-object pixel similarity and inter-object dissimilarity, improving mask completeness. The architecture leverages Transformer-based encoders for efficient multi-scale feature fusion, outperforming existing single-level alignment models in accuracy and robustness.
Novelty
This is the first work to explicitly integrate sentence-mask alignment with word-pixel alignment hierarchically, capturing intra-object pixel coherence and spatial relationships simultaneously. Unlike prior models focusing solely on either pixel-level or sentence-level alignment, CoupAlign’s multi-level approach effectively mitigates fragmentation and boundary errors, especially in multi-object and complex scene scenarios, representing a significant methodological leap.
Limitations
- Despite improvements, the model still struggles with occlusion and highly blurred boundaries, where pixel coherence is ambiguous. Computational complexity from multi-scale attention and mask proposals limits real-time deployment.
- Handling very long or complex sentences remains challenging, as the current sentence embedding may not fully capture nuanced semantics. Future work should explore more efficient architectures and richer language understanding.
- The reliance on pre-trained encoders like Swin Transformer and BERT, while effective, introduces dependency on large-scale pretraining and may limit adaptation to domain-specific tasks without further fine-tuning.
Future Work
Future directions include optimizing the computational efficiency of hierarchical alignment, integrating self-supervised learning to reduce reliance on large pretraining, and extending the framework to video and 3D scene understanding. Enhancing semantic parsing for complex sentences and incorporating contextual reasoning will further improve robustness and generalization, broadening the applicability of RIS in real-world scenarios.
AI Executive Summary
Referring image segmentation (RIS) has become a crucial task in multimodal understanding, aiming to localize all pixels of objects described by natural language. Existing methods largely rely on either sentence-pixel or early fusion strategies, which often lead to fragmented masks and localization errors, especially when multiple similar objects are involved. These limitations hinder practical applications such as image editing, autonomous navigation, and robotic interaction, where precise and complete object masks are essential.
CoupAlign introduces a hierarchical multi-level alignment framework that couples sentence-mask and word-pixel alignments to address these challenges. The core idea is to enforce object mask constraints at both visual and semantic levels, ensuring pixel coherence within objects and accurate localization. The architecture integrates a Swin Transformer-based image encoder and a BERT-based language encoder, with a Word-Pixel Alignment (WPA) module performing early, bidirectional cross-attention at intermediate layers. This module fuses linguistic and visual features, generating aligned embeddings from which multiple mask proposals are derived.
Subsequently, the Sentence-Mask Alignment (SMA) module weights these masks based on sentence embeddings, refining the localization of the referred object. The masks are projected back into pixel space, guided by hierarchical features, to produce precise segmentation maps. An auxiliary contrastive loss further enhances pixel-wise semantic consistency by pulling foreground pixels closer and pushing background pixels apart, reducing fragmentation and boundary errors.
Extensive experiments on datasets like RefCOCO, RefCOCO+, and G-Ref demonstrate that CoupAlign outperforms state-of-the-art methods, achieving approximately 2% higher oIoU scores. It excels in complex scenarios involving multiple similar objects, accurately distinguishing targets even in crowded scenes. The results validate that hierarchical, multi-level alignment significantly improves mask completeness and localization accuracy.
This work marks a substantial step forward in multimodal scene understanding, combining innovative architecture design with effective training strategies. Its ability to generate more accurate, complete masks opens new possibilities for real-world applications requiring fine-grained image segmentation guided by natural language. Future research will focus on reducing computational costs, extending to video and 3D data, and enhancing semantic reasoning capabilities, aiming for more intelligent and versatile visual understanding systems.
Deep Dive
Abstract
Referring image segmentation aims at localizing all pixels of the visual objects described by a natural language sentence. Previous works learn to straightforwardly align the sentence embedding and pixel-level embedding for highlighting the referred objects, but ignore the semantic consistency of pixels within the same object, leading to incomplete masks and localization errors in predictions. To tackle this problem, we propose CoupAlign, a simple yet effective multi-level visual-semantic alignment method, to couple sentence-mask alignment with word-pixel alignment to enforce object mask constraint for achieving more accurate localization and segmentation. Specifically, the Word-Pixel Alignment (WPA) module performs early fusion of linguistic and pixel-level features in intermediate layers of the vision and language encoders. Based on the word-pixel aligned embedding, a set of mask proposals are generated to hypothesize possible objects. Then in the Sentence-Mask Alignment (SMA) module, the masks are weighted by the sentence embedding to localize the referred object, and finally projected back to aggregate the pixels for the target. To further enhance the learning of the two alignment modules, an auxiliary loss is designed to contrast the foreground and background pixels. By hierarchically aligning pixels and masks with linguistic features, our CoupAlign captures the pixel coherence at both visual and semantic levels, thus generating more accurate predictions. Extensive experiments on popular datasets (e.g., RefCOCO and G-Ref) show that our method achieves consistent improvements over state-of-the-art methods, e.g., about 2% oIoU increase on the validation and testing set of RefCOCO. Especially, CoupAlign has remarkable ability in distinguishing the target from multiple objects of the same class.