UFVideo: Towards Unified Fine-Grained Video Cooperative Understanding with Large Language Models

Video understanding has evolved from early feature-based methods like C3D and I3D to deep learning models such as VideoBERT and VideoLLaMA, which leverage multimodal data. Recent works like RGA3 and UniPixel have focused on pixel-level tasks, but they operate in isolation, limiting their ability to perform multi-scale understanding. The challenge lies in integrating global, pixel, and temporal information into a cohesive model that can handle complex, real-world scenarios. Multi-task learning frameworks have been proposed, but they often lack effective mechanisms for cross-scale fusion, resulting in fragmented perception capabilities.

Current models are specialized for single granularities, such as object referring or temporal grounding, leading to limited understanding in complex scenes requiring multiple perspectives. The absence of a unified framework hampers the ability to perform comprehensive video analysis, especially when multiple tasks need to be addressed simultaneously. This fragmentation reduces efficiency and accuracy, constraining practical applications like video summarization, detailed QA, and interactive segmentation. Developing a model that can seamlessly integrate these tasks remains a core challenge.

UFVideo introduces a multi-scale unified framework that combines global, pixel, and temporal understanding within a single architecture. It employs special tokens to differentiate task types, enabling flexible task-specific processing. The core innovation is the visual-language guided alignment, which aligns features across scales, and the integration of SAM2 decoders for pixel-level segmentation. Multi-stage training strategies further enhance task synergy, allowing the model to learn shared representations that support diverse tasks simultaneously. This approach surpasses prior models by enabling true multi-granular understanding, bridging the gap between isolated task-specific models.

• �� Encode videos using a pre-trained vision encoder (e.g., SigLIP) to extract visual features. • Insert special tokens (<Temp>, <Ref>, <Seg>) into textual prompts to specify task granularity. • Use a unified visual-language encoder to align visual features and text tokens, enabling cross-modal understanding. • For object referring, inject object visual prompts and encode spatial features, then map to tokens for the LLM. • For segmentation, select key frames, encode with SAM2, and generate pixel masks conditioned on language embeddings. • During training, optimize next-token prediction for language output, combined with BCE and Dice losses for masks. • Multi-stage training first aligns temporal and segmentation tasks, then jointly fine-tunes for all tasks, ensuring multi-scale coherence.

The model was evaluated on nine public benchmarks, including MVBench, VideoRefer-Bench, and others, measuring accuracy, precision, and recall. Hyperparameters like learning rate (2e-5), batch size (512/256), and training epochs (2/1) were tuned for optimal performance. Ablation studies confirmed the importance of special tokens and multi-stage training. The model demonstrated superior performance in multi-task scenarios, with significant gains over previous models, especially in pixel segmentation and temporal localization tasks. Cross-scenario robustness was validated through diverse datasets and ablation experiments.

UFVideo achieved an average score of 67.3% across nine benchmarks, outperforming models like LLaVA-ST and UniPixel by 5-7%. In pixel-level segmentation, accuracy improved by 12%, and temporal localization accuracy increased by 15%. The model's multi-task capability was validated by its consistent performance across diverse tasks, confirming the effectiveness of the multi-scale fusion strategy. These results demonstrate the model’s ability to understand videos at multiple granularities simultaneously, setting new state-of-the-art standards.

This technology can be directly applied in automated video analysis, surveillance, content moderation, and virtual assistants, enabling detailed scene understanding and interaction. It supports complex queries involving object references, scene segmentation, and event timing, facilitating smarter AI systems. Long-term, integrating such models with edge devices and real-time processing will revolutionize industries like security, entertainment, and autonomous systems.

Despite its strengths, UFVideo requires substantial computational resources, limiting deployment in resource-constrained environments. Handling very long videos or high frame-rate streams remains challenging, with performance drops in temporal localization. Robustness under noisy, occluded, or highly dynamic scenes needs further improvement. Future work should focus on efficiency, scalability, and robustness to broaden practical usability.