MobileVOS: Real-Time Video Object Segmentation Contrastive Learning meets Knowledge Distillation

Video object segmentation has evolved from early template-matching and handcrafted feature methods to deep learning-based approaches like OSVOS, STM, and XMem, which leverage temporal memory and feature matching. While these models achieve high accuracy, their large parameter count and computational complexity hinder deployment on resource-limited devices. Recent efforts focus on model compression, lightweight backbones (e.g., MobileNet), and efficient memory management, yet a gap remains in achieving real-time performance on mobile hardware without sacrificing accuracy. The challenge lies in balancing model size, speed, and robustness, especially in complex scenes with occlusion and boundary ambiguity.

Existing high-performance models are too resource-intensive for mobile deployment, limiting real-time applications. Limited memory networks struggle with boundary precision and occlusion, and architectural complexity increases energy consumption. The core issue is how to effectively transfer rich structural information from large models to small, fast networks, ensuring accurate boundary delineation and temporal consistency under strict computational constraints. Addressing this requires innovative loss functions and training strategies that can enhance feature discriminability and structural fidelity within limited parameters.

This work introduces a novel pixel-wise structure distillation loss based on mutual information maximization, enabling effective structural knowledge transfer from teacher to student. It combines supervised contrastive learning with boundary-aware sampling, focusing training on boundary pixels to improve detail accuracy. The framework simplifies architecture requirements, avoiding complex memory modules, and achieves significant parameter reduction (32x) while boosting inference speed (5x). The theoretical grounding in information theory provides robustness and generality, making it adaptable to various dense prediction tasks. This approach marks a shift from architectural complexity to loss function innovation for model compression.

• �� Teacher-student architecture with the teacher having infinite memory and the student limited to recent frames. • Construction of pixel-wise correlation matrices (Cs, Ct) from normalized feature representations to encode structural relationships. • Design of a structure-related loss (Lrepr) that maximizes mutual information between student and teacher features, regularized by correlation alignment. • Incorporation of boundary-aware sampling, which selects pixels near object edges using Sobel edge detection, reducing computational load and emphasizing boundary details. • Integration of supervised contrastive loss (LSupCon) by interpolating the target correlation matrix with ground-truth labels, controlled by a hyperparameter ω. • Combining structure loss, contrastive loss, and logit distillation (KL divergence) into a unified multi-objective training pipeline. • End-to-end training with pre-trained teacher models, optimizing for accuracy and efficiency.

The models are trained on DAVIS 2016/2017 and YouTube-VOS datasets, using a pre-trained STCN as teacher. Architectures include MobileNetV2 and ResNet18 backbones, with boundary sampling and multi-loss optimization. Evaluation metrics include J&F, FPS, and parameter count, tested on NVIDIA A40, 1080Ti, and mobile GPU. Ablation studies verify the contribution of each component, showing structure loss and boundary sampling significantly improve boundary accuracy and convergence. The models are deployed on a Samsung Galaxy S22, demonstrating real-time inference at over 90 FPS with minimal latency. Model soups are used to enhance robustness and performance stability across different scenarios.

MobileVOS ResNet18 achieves J&F of 90.4 on DAVIS 2016, with 4x speed and 3x fewer parameters compared to STCN. On DAVIS 2017, J&F reaches 89.7, close to the teacher, with 5x faster inference. On YouTube-VOS, the model maintains high accuracy at 90 FPS, with parameters only 1/4 of competing methods. These results confirm the effectiveness of the proposed loss and training strategy, enabling high-quality, real-time segmentation on mobile devices while reducing computational costs dramatically.

This approach enables real-time video analysis on smartphones, drones, and embedded systems, facilitating applications like live video editing, augmented reality, and surveillance. Its low computational footprint makes it suitable for edge AI, reducing reliance on cloud processing. Future integration with multi-modal sensors (depth, optical flow) could further enhance robustness, supporting broader deployment in autonomous vehicles, smart cameras, and wearable devices, transforming mobile video understanding.

Despite significant progress, boundary prediction errors under severe occlusion or rapid motion remain. Dependence on teacher model quality limits performance if the teacher is weak. The boundary sampling strategy may miss some critical pixels in highly complex scenes. Additionally, further energy-efficient hardware-aware design is needed for sustained operation on ultra-low-power devices. Future work should address these issues by exploring adaptive sampling, multi-modal fusion, and hardware-specific optimizations to improve robustness and efficiency in diverse real-world scenarios.