Look Before You Match: Instance Understanding Matters in Video Object Segmentation

VOS evolved from optical-flow propagation and online adaptation toward memory matching, exemplified by STM, STCN, and XMem. Propagation is vulnerable to occlusion and error accumulation; memory methods capture longer temporal context and therefore dominate recent benchmarks. However, dense feature matching mainly asks whether two regions look alike. It does not explicitly represent which instance owns the pixels, leaving a weakness under deformation, occlusion, and multi-object confusion on DAVIS and YouTube-VOS.

Given a first-frame target mask, the model must preserve object identity and predict pixel masks in later frames. The target may change pose, scale, viewpoint, visibility, or illumination, while another object may share its color and texture. Pure historical retrieval provides unreliable correspondences for newly visible or severely deformed regions, producing identity switches, false matches, and inaccurate boundaries.

First, ISVOS learns instance-discriminative features with query-based segmentation rather than relying only on pixel similarity. Second, Enhanced Key Encoder uses Deformable Attention to inject object queries into the query key for instance-augmented matching. Third, MPF combines memory readout with backbone and pixel-decoder features to restore fine boundaries. Fourth, the branches are jointly trained, but instance mask outputs are discarded during inference; only useful intermediate representations remain.

• �� The current frame Xt is processed by ResNet to obtain Fres4; a pixel decoder produces Fpixel and multiscale {Pi}.
• �� A Transformer decoder uses 100 learnable object queries and masked attention, ql=softmax(Ml−1+qlkᵀ)vl+ql−1, to obtain instance queries.
• �� Enhanced Key Encoder maps Fres4 to Qg, applies DeformAttn(q̃ins,p,Qg), and constructs an instance-aware query key Q.
• �� Memory keys K and values V are retrieved. Affinity is Ai,j=exp(d(Ki,Qj))/Σi exp(d(Ki,Qj)), where d is normalized L2 distance; weighted values produce Fmem.
• �� MPF progressively fuses Fmem with backbone features Bi and pixel features Pi, producing a stride-4 mask that is upsampled. Inference stores every fifth frame in a FIFO bank of size 16.

The IS branch is trained on COCO; the VOS branch uses deformed static images, YouTube-VOS, and DAVIS, with optional BL30K pretraining. AdamW is used; static pretraining lasts 150K iterations at 4e−5, and main training lasts 110K iterations at 2e−5. Evaluation covers DAVIS 2016/2017 and YouTube-VOS 2018/2019 using J, boundary F, and J&F. Baselines include STM, STCN, XMem, AOT, DeAOT, and RDE. Ablations remove Query Enhancement, MPF, Mask2Former initialization, or joint training.

Without BL30K, ISVOS scores 92.6% on DAVIS16 val, 87.1% on DAVIS17 val, and 86.3% on YT2018 val, compared with XMem's 91.5%, 86.2%, and 85.7%. It reaches 82.8% on DAVIS17 test-dev and 86.1% on YT2019 val; with BL30K, these become 84.0% and 86.3%. Removing QE costs 1.4 and 1.9 J&F points on DAVIS17 and YT2018, confirming that instance-conditioned matching is central.

The framework can support interactive video editing, object-aware tracking, autonomous driving, and sports analysis. A deployment needs an initial mask or user prompt, a stable video stream, and memory-aware acceleration. Because it requires no post-processing and uses a bounded memory queue, it is attractive for applications that need both identity continuity and accurate contours.

The method assumes a reliable first-frame annotation and benefits from COCO-pretrained instance representations; large domain or category shifts may weaken them. Fixed FIFO storage and every-fifth-frame updates may not suit all motion patterns or video lengths, and extreme occlusion can still cause drift. Future systems should use adaptive memory, open-domain instance representations, efficient inference, and uncertainty-triggered human correction.