ReVSeg: Incentivizing the Reasoning Chain for Video Segmentation with Reinforcement Learning

Video object segmentation (VOS) has evolved from early appearance-based methods like OSVOS and MaskTrack to deep learning models utilizing CNNs and Transformers. Recently, vision-language models (VLMs) such as CLIP and Flamingo have demonstrated strong cross-modal understanding, inspiring their application in VOS. However, most existing approaches rely on single-step predictions or latent embeddings, which lack transparency and struggle with complex reasoning involving causality, dynamics, and abstract instructions. These limitations hinder the deployment of VOS in real-world scenarios requiring interpretability and robustness.

The core challenge is enabling models to perform multi-faceted reasoning—understanding scene dynamics, causal relationships, and temporal interactions—beyond static appearance cues. Current models often treat reasoning as a monolithic prediction, leading to opaque decision processes and poor generalization. Moreover, annotated data for explicit reasoning is scarce, complicating supervision. How to structure reasoning as an explicit, optimizable process that aligns with pretrained capabilities remains an open problem, crucial for advancing video understanding in complex, real-world environments.

ReVSeg introduces a structured, multi-step reasoning framework that decomposes complex video understanding into primitive operations aligned with VLM capabilities. It replaces single-step latent prediction with explicit semantics interpretation, temporal evidence selection, and spatial grounding, executed via multi-turn interactions. The integration of reinforcement learning (GRPO) to reward intermediate decisions is a key innovation, enabling the model to self-improve without dense supervision. This approach enhances transparency, robustness, and performance, representing a paradigm shift from traditional end-to-end models.

• �� Input video and query, perform semantics interpretation to understand scene and generate target description. • Select keyframes based on scene dynamics and query relevance, using temporal grounding. • In subsequent step, localize target object within the keyframe via spatial grounding, producing bounding boxes. • Use a single VLM (Qwen2.5-VL-7B) to execute all steps, maintaining semantic consistency. • Employ reinforcement learning with sparse rewards, designed to evaluate format correctness, frame quality, and localization accuracy. • Optimize the policy via GRPO, sampling multiple candidates, evaluating rewards, and updating the decision strategy iteratively. • During training, utilize datasets like Ref-YouTube-VOS, ReVOS, and custom reward signals to improve reasoning chain quality.

The framework was evaluated on ReasonVOS, ReVOS, and referring VOS datasets, comparing against SOTA models such as RGA-7B, GLUS, and CoT-RVS. Metrics included J&F scores, with ReVSeg-7B achieving 64.8 on ReasonVOS, outperforming baselines by over 10 points. Ablation studies confirmed the importance of explicit reasoning decomposition and reinforcement learning. Hyperparameters included a learning rate of 1e-6, batch size 128, and 8 rollouts per prompt. The training process involved filtering data for quality and using reward signals at key decision points, demonstrating improved robustness and interpretability across scenarios.

ReVSeg outperformed all baselines, achieving 64.8 J&F on ReasonVOS, 62.1 on ReVOS, and setting new standards on referring datasets. Ablation results showed that removing reasoning steps or reinforcement learning reduced performance by significant margins, confirming their critical roles. The model's ability to generate interpretable reasoning trajectories was validated through qualitative analysis, illustrating how explicit decision chains improve understanding and debugging. These results underscore the effectiveness of structured reasoning combined with reinforcement learning in complex video tasks.

This approach can be directly applied to autonomous driving for precise object tracking and scene understanding, enhancing safety and decision-making. In surveillance, it enables robust tracking of suspicious activities with explainable reasoning. In human-computer interaction, it facilitates more natural and interpretable AI systems capable of understanding complex instructions. The framework's modular design allows adaptation to other vision tasks like video question answering and scene graph generation, broadening its industrial impact.

Despite strong results, the model faces challenges with severe occlusion and rapid motion, where decision errors propagate. The reinforcement learning training is computationally intensive, requiring extensive interaction data. Dependence on pretrained models limits flexibility across domains, necessitating domain-specific fine-tuning. Future work should focus on improving efficiency, reducing reliance on large-scale pretraining, and enhancing robustness in extreme scenarios.