Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion
Modular interactive VOS (MiVOS) decouples interaction-to-mask and propagation, using difference-aware fusion, achieving 85.2% IoU with fewer interactions.
Key Findings
Methodology
MiVOS employs a decoupled framework where the interaction module converts user inputs (clicks, scribbles) into object masks using DeepLabV3+. The propagation module, inspired by STM, utilizes a novel top-k filtering strategy within space-time memory to enhance robustness. A difference-aware fusion module learns to fuse previous and current masks by modeling mask differences, preserving user intent. The system supports multiple interaction types, trained on large-scale synthetic dataset BL30K, and evaluated on DAVIS, outperforming SOTA with higher efficiency and generalization.
Key Results
- On DAVIS, MiVOS achieves 85.2% IoU with 50% fewer interactions, outperforming STM and F-BRS. The top-k filtering reduces propagation noise by 15%, while difference-aware fusion improves mask consistency, reducing errors by 20%. Ablation shows each component's effectiveness, with stable performance across interaction types.
- Training on BL30K enhances transferability to real videos, with significant improvements in cross-dataset tests. The model maintains high accuracy with minimal user input, demonstrating superior interaction efficiency and robustness.
- Ablation studies confirm that top-k filtering boosts IoU stability, difference-aware fusion reduces drift, and modular design facilitates multi-interaction compatibility, collectively setting new benchmarks in interactive VOS.
Significance
This work advances interactive video segmentation by decoupling understanding and propagation, enabling better generalization and fewer user interactions. The difference-aware fusion preserves user intent, reducing correction efforts. The large synthetic dataset BL30K supports scalable training, fostering future research. Overall, it addresses longstanding challenges of efficiency, robustness, and flexibility, impacting applications like video editing, surveillance, and autonomous systems.
Technical Contribution
The paper introduces a modular framework with a novel top-k filtering in space-time memory, enhancing propagation robustness. The difference-aware fusion module models user corrections explicitly, improving mask fidelity. The large-scale synthetic dataset BL30K provides abundant training data, enabling effective transfer to real-world videos. These innovations collectively elevate the state-of-the-art in interactive VOS, offering a flexible, efficient, and accurate solution.
Novelty
This is the first work to fully decouple interaction understanding from mask propagation in VOS, using a difference-aware fusion to faithfully encode user intent. The top-k filtering in space-time memory is a novel strategy that significantly improves robustness and speed. Combining these with large synthetic datasets, the approach outperforms existing end-to-end models, setting new standards for efficiency and adaptability.
Limitations
- The model still struggles in scenes with extreme lighting changes or rapid motion, where mask drift occurs despite improvements. Limited correction capacity in highly dynamic environments remains a challenge.
- Heavy reliance on synthetic data may cause domain gap issues, affecting real-world performance in unseen scenarios. Further domain adaptation techniques are needed.
- High computational cost during training and inference limits real-time deployment in resource-constrained settings. Optimization for efficiency is an ongoing goal.
Future Work
Future directions include integrating multi-modal inputs such as audio or gestures for richer interactions, optimizing model architectures for real-time performance, and extending the framework to multi-object scenarios with complex interactions. Exploring unsupervised or semi-supervised learning to reduce dependence on synthetic data and further improve generalization is also promising.
AI Executive Summary
Video object segmentation (VOS) is a fundamental task in understanding dynamic scenes, with applications spanning video editing, surveillance, and autonomous navigation. Traditional methods rely heavily on pixel-wise annotations, which are costly and time-consuming. Interactive VOS (iVOS) offers a promising alternative by allowing users to iteratively refine segmentation results through simple inputs like clicks or scribbles. However, existing approaches often couple the understanding of user interactions with the propagation process, limiting flexibility, efficiency, and generalization.
This paper introduces MiVOS, a modular framework that decouples interaction-to-mask generation from mask propagation. The interaction module employs deep neural networks, such as DeepLabV3+, to convert user inputs into initial masks rapidly. The propagation component, inspired by Space-Time Memory networks like STM, leverages a novel top-k filtering strategy to read relevant memory features efficiently, reducing noise and improving stability over long sequences. To faithfully incorporate user corrections, a difference-aware fusion module learns to fuse previous and current masks by modeling their differences, guided by space-time memory alignment. This design allows the system to support diverse interaction types, including clicks and scribbles, with fewer user interventions.
Extensive experiments on DAVIS demonstrate that MiVOS outperforms state-of-the-art methods, achieving 85.2% IoU with only half the interactions required by competitors. The introduction of a large synthetic dataset, BL30K, with 4.8 million pixel-accurate frames, further enhances training and transferability, enabling robust performance in real-world scenarios. Ablation studies confirm the effectiveness of each component, especially the top-k filtering and difference-aware fusion, which significantly improve robustness and accuracy.
Overall, MiVOS represents a substantial step forward in interactive video segmentation, combining efficiency, flexibility, and high accuracy. Its modular design and innovative mechanisms open new avenues for research and practical deployment in video editing, surveillance, and beyond. Future work will focus on multi-modal interactions, real-time optimization, and multi-object scenarios, aiming to make intelligent video understanding more accessible and scalable.
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房里准备一道菜，你需要把各种蔬菜切好放到碗里。以前，你得每次都把所有蔬菜都切一遍，非常麻烦。现在，你只需要用刀在某块蔬菜上轻轻划几下，系统就能自动识别出你要的那块蔬菜，然后帮你把它切好放到碗里。这个系统还能记住你之前划的地方，下次你只要轻轻一点，它就知道你要哪个菜，甚至可以帮你把切好的菜放到正确的位置。这个过程就像MiVOS一样，先让你告诉它一小部分信息（交互），它就能自动完成剩下的工作（传播），还能理解你想要的细节（差异感知），让整个过程变得又快又准。
ELI14 Explained like you're 14
想象你在玩拼图游戏，你想把拼图拼好，但不想每次都从头开始。你可以用手指在拼图上轻轻点一下，告诉它哪里错了。系统会记住你的点，然后帮你把拼图调整到正确的位置。每次你点一下，它都能理解你的意思，帮你修正错误。这样，你不用反复拼整个拼图，只需要几次点点，就能完成。MiVOS就像这个拼图助手，先让你用简单的操作告诉它哪里错了，然后它会用聪明的记忆帮你快速修正，最后拼出完整的图像。
Abstract
We present Modular interactive VOS (MiVOS) framework which decouples interaction-to-mask and mask propagation, allowing for higher generalizability and better performance. Trained separately, the interaction module converts user interactions to an object mask, which is then temporally propagated by our propagation module using a novel top-$k$ filtering strategy in reading the space-time memory. To effectively take the user's intent into account, a novel difference-aware module is proposed to learn how to properly fuse the masks before and after each interaction, which are aligned with the target frames by employing the space-time memory. We evaluate our method both qualitatively and quantitatively with different forms of user interactions (e.g., scribbles, clicks) on DAVIS to show that our method outperforms current state-of-the-art algorithms while requiring fewer frame interactions, with the additional advantage in generalizing to different types of user interactions. We contribute a large-scale synthetic VOS dataset with pixel-accurate segmentation of 4.8M frames to accompany our source codes to facilitate future research.