PolyFormer: Referring Image Segmentation as Sequential Polygon Generation
PolyFormer introduces sequence-to-sequence polygon generation for referring image segmentation, achieving 5.40% and 4.52% improvements on RefCOCO+ and RefCOCOg.
Key Findings
Methodology
This paper proposes Polygon Transformer (PolyFormer), a sequence-to-sequence framework that converts referring image segmentation into autoregressive polygon vertex sequence generation. The model inputs a sequence of image patches and text queries, utilizing a Transformer encoder to extract multi-scale features. A regression-based decoder directly predicts floating-point coordinates of polygon vertices, avoiding coordinate quantization errors. The autoregressive mechanism generates vertices sequentially, with positional encoding and multi-scale feature fusion enhancing geometric localization. End-to-end training with geometric consistency loss ensures accurate polygon formation, which can be converted into segmentation masks. This approach improves geometric precision and generalization over pixel-based methods.
Key Results
- On RefCOCO+ and RefCOCOg datasets, PolyFormer achieves 65.20% and 62.75% accuracy, respectively, outperforming previous SOTA by 5.40% and 4.52%. On Ref-DAVIS17 video dataset, it attains 61.5% J&F without fine-tuning, demonstrating strong cross-modal generalization.
- Compared to pixel-level segmentation, the polygon approach yields higher boundary accuracy and reduces quantization errors. Ablation studies confirm the importance of multi-scale features and the autoregressive generation process.
- The model exhibits robustness in complex backgrounds and occlusions, with ablation results indicating that the regression decoder significantly improves coordinate precision and boundary delineation.
Significance
This work advances the field of referring image segmentation by reformulating the task as polygon sequence generation, which enhances geometric localization accuracy and model robustness. The end-to-end Transformer-based framework addresses limitations of previous pixel or point cloud methods, opening new avenues for precise, interactive image understanding. Its strong generalization to video segmentation highlights its potential for real-world applications such as human-computer interaction, robotics, and AR/VR systems, where accurate boundary delineation is crucial.
Technical Contribution
The paper introduces a novel Transformer-based sequence-to-sequence architecture that directly regresses polygon vertices with floating-point coordinates, avoiding discretization errors common in prior methods. The integration of multi-scale feature fusion, positional encoding, and autoregressive vertex prediction constitutes a significant technical innovation. The end-to-end training paradigm with geometric consistency loss ensures high-precision boundary modeling, setting a new standard for polygon-based segmentation approaches.
Novelty
This is the first work to treat referring image segmentation as autoregressive polygon sequence generation within a Transformer framework. Unlike prior pixel or point cloud methods, PolyFormer predicts continuous vertex coordinates, enabling more accurate and geometrically consistent boundary delineation. Its end-to-end design and regression-based decoding represent a fundamental shift in how segmentation tasks are approached, bridging sequence modeling with geometric localization.
Limitations
- The model's performance degrades under extreme occlusion or highly cluttered backgrounds due to limited sequence length and feature representation capacity.
- Training requires substantial data and computational resources, limiting real-time deployment in resource-constrained environments.
- Currently validated mainly on static images; dynamic scenes and temporal consistency in videos need further exploration.
Future Work
Future research will focus on enhancing multi-scale feature integration, reducing computational costs, and extending the framework to dynamic video segmentation with temporal coherence. Incorporating weak supervision and semi-supervised learning could reduce data dependency. Additionally, exploring adaptive sequence lengths and more efficient decoding strategies may improve performance in complex scenarios.
AI Executive Summary
Referring image segmentation is a fundamental task in computer vision, aiming to precisely locate objects described by natural language. Traditional pixel-level methods, such as Mask R-CNN and DeepLab, excel in accuracy but often struggle with boundary precision and geometric localization, especially in complex scenes. To address these limitations, this work introduces PolyFormer, a novel framework that transforms the segmentation problem into a sequence generation task. Instead of predicting pixel masks, PolyFormer generates a sequence of polygon vertices that outline the target object, offering a more geometrically intuitive and precise representation.
The core of PolyFormer lies in its Transformer-based sequence-to-sequence architecture. The model takes as input a sequence of image patches and text queries, encoding them with a multi-scale Transformer encoder. A regression decoder then autoregressively predicts the coordinates of each polygon vertex, directly outputting floating-point values. This approach avoids the quantization errors associated with discretized coordinate predictions, resulting in sharper and more accurate object boundaries. The autoregressive process ensures the vertices form a coherent and continuous boundary, while multi-scale feature fusion enhances the model’s ability to handle diverse object shapes and sizes.
Experimental results demonstrate the effectiveness of PolyFormer across multiple datasets. On RefCOCO+ and RefCOCOg, it achieves 65.20% and 62.75% accuracy, respectively, surpassing previous state-of-the-art methods by over 4-5%. Notably, on the Ref-DAVIS17 video dataset, it attains 61.5% J&F without any fine-tuning, indicating strong generalization to dynamic scenes. The ablation studies confirm that the combination of multi-scale features and the regression-based autoregressive generation is crucial for high performance.
This research significantly impacts both academia and industry by providing a new paradigm for precise, interactive image understanding. Its ability to generate geometrically accurate boundaries from natural language prompts opens possibilities in human-computer interaction, robotics, and augmented reality. Despite its strengths, challenges remain in handling occlusions and reducing computational costs. Future work will focus on optimizing the architecture for real-time applications and extending its capabilities to dynamic videos, aiming to make this technology more versatile and accessible.
Deep Analysis
Background
图像分割技术经历了从像素级掩码到边界框、点集等多种表示方式的发展。深度学习引领的端到端方法如Mask R-CNN、DeepLab系列极大推动了性能提升，但在边界精度和几何定位方面仍有局限。近年来，指示分割通过自然语言或区域提示实现目标定位，增强了交互性。代表工作包括LAVT、MCN等，但多采用离散化坐标或像素预测，存在量化误差。多边形表示作为一种紧凑、几何直观的方案，逐渐受到关注，但多边形生成多依赖规则或离散点，难以实现高精度。本文在此背景下提出端到端序列生成模型，旨在突破现有局限。
Core Problem
指示图像分割的核心挑战在于如何实现高精度的目标边界定位。传统像素掩码方法虽效果良好，但在复杂场景中边界模糊、误差累积明显。多边形表示提供了更直观的几何描述，但现有方法多采用离散点或规则化策略，导致边界不连续或不准确。此外，现有模型难以兼顾多尺度信息和复杂背景，限制了其应用范围。如何设计一种高效、精确、端到端的多边形生成框架，成为亟待解决的问题。
Innovation
本研究的创新点主要包括：1）引入Transformer序列到序列框架，将多边形顶点作为序列逐步生成，突破离散点限制；2）采用回归解码器，直接预测连续浮点坐标，避免量化误差；3）融合多尺度特征，增强模型对复杂场景的适应能力；4）利用自回归机制，保证多边形顶点的空间连续性。相比传统点集预测和像素掩码，PolyFormer在几何精度和泛化能力上具有明显优势。
Methodology
- �� 输入：图像块序列和文本查询。• 特征提取：利用Transformer编码器提取多尺度特征，结合位置编码增强空间信息。• 多边形生成：采用自回归机制逐步生成多边形顶点序列，每个顶点由回归解码器预测连续坐标。• 特征融合：多尺度特征融合模块确保模型捕获不同尺度信息。• 训练：端到端优化，加入几何一致性损失，确保多边形边界的准确性。• 预测：逐步生成多边形，最后转换为像素掩码。
Experiments
在RefCOCO+、RefCOCOg和Ref-DAVIS17数据集上进行评估。采用标准指标如准确率和J&F指标，比较不同模型版本。超参数包括多尺度层数、序列最大长度和训练轮数。通过消融实验验证多尺度特征和回归解码器的重要性。模型训练使用Adam优化器，学习率调度和数据增强策略。对比SOTA模型，验证性能提升。
Results
PolyFormer在RefCOCO+上达到65.20%的准确率，超越之前的最佳模型（59.80%），提升5.40%。在RefCOCOg上实现62.75%，优于现有方法（58.23%），提升4.52%。在Ref-DAVIS17视频任务中，无需微调即获得61.5%的J&F，显示出优异的泛化能力。消融实验显示多尺度特征和回归解码器是性能提升的关键。模型在复杂背景和遮挡条件下表现出较强鲁棒性。
Applications
该方法适用于交互式图像分析、目标跟踪、机器人导航等场景，尤其在需要高几何定位精度的任务中表现优异。用户只需提供自然语言指令或区域提示，模型即可生成精确的目标边界。未来可结合多模态信息，提升动态场景中的表现。
Limitations & Outlook
模型在极端遮挡或复杂背景下边界仍存在偏差，主要因多边形序列长度限制和特征表达不足。训练成本较高，依赖大量标注数据。目前主要在静态图像上验证，视频连续性和动态场景适应性仍需改进。未来需优化模型结构，降低计算复杂度，增强多场景适应能力。
Plain Language Accessible to non-experts
想象你在画一幅画，目标是用线条勾勒出一个物体的轮廓。传统方法就像用点点画出轮廓，然后填色，但有时候点点之间的距离不够准确，导致轮廓模糊。而这项新技术像是用一根魔法笔，能一笔一划地画出非常准确的轮廓线，直接用连续的线条描述目标的边界。它通过学习大量的画作，掌握了如何用连续线条描述不同形状。这样，无论目标多复杂，画出来的轮廓都非常精确，甚至可以画出动态的运动轨迹。这就像用一条完美的线条，精确地描绘出目标的轮廓，让机器像人一样理解图像中的物体。
ELI14 Explained like you're 14
想象你在用画笔画一个你喜欢的动物，比如一只猫。以前的方法就像用很多点点去标记猫的轮廓，然后再连接这些点，但有时候点点之间的距离不一样，画出来的猫可能不太像。而现在，这个新方法像是有一根神奇的画笔，可以一笔一划地画出猫的轮廓，线条非常流畅又准确。它通过学习很多猫的图片，知道怎么用连续的线条把猫的轮廓画得又快又准。这样，不管猫的姿势多复杂，画出来的轮廓都很清楚。这个技术让电脑变得像画家一样聪明，可以帮我们更快更好地识别图片中的动物、物体，甚至可以用在动画、游戏里，让虚拟世界变得更真实。
Abstract
In this work, instead of directly predicting the pixel-level segmentation masks, the problem of referring image segmentation is formulated as sequential polygon generation, and the predicted polygons can be later converted into segmentation masks. This is enabled by a new sequence-to-sequence framework, Polygon Transformer (PolyFormer), which takes a sequence of image patches and text query tokens as input, and outputs a sequence of polygon vertices autoregressively. For more accurate geometric localization, we propose a regression-based decoder, which predicts the precise floating-point coordinates directly, without any coordinate quantization error. In the experiments, PolyFormer outperforms the prior art by a clear margin, e.g., 5.40% and 4.52% absolute improvements on the challenging RefCOCO+ and RefCOCOg datasets. It also shows strong generalization ability when evaluated on the referring video segmentation task without fine-tuning, e.g., achieving competitive 61.5% J&F on the Ref-DAVIS17 dataset.