Valley: Video Assistant with Large Language model Enhanced abilitY

The evolution of multimodal learning has transitioned from early image-text models like CLIP and ViT to sophisticated video understanding frameworks. Early works focused on static image-text alignment, but long videos pose challenges due to high dimensionality and temporal dependencies. Datasets like WebVid and ActivityNet provided foundational data, yet suffered from quality and diversity issues. Recent efforts, including LLaVA and MiniGPT-4, introduced instruction tuning to improve visual-language interaction. However, these models primarily target short videos or images, leaving long video comprehension underdeveloped. The complexity of temporal dynamics and multi-task demands necessitate advanced architectures that can efficiently model long-range dependencies and diverse content. Addressing these gaps, this paper proposes a comprehensive framework that combines high-capacity vision encoders, multiple temporal modules, and large-scale curated datasets, aiming to push the boundaries of video understanding.

Current multimodal models struggle with effectively understanding long videos due to limitations in temporal feature extraction and data quality. Existing datasets lack sufficient diversity and temporal richness, leading to suboptimal generalization. Moreover, integrating visual and textual modalities in a way that supports complex instructions remains challenging. These issues hinder the deployment of AI assistants capable of nuanced video comprehension, such as causal inference, long-form captioning, and multi-shot reasoning. Overcoming these bottlenecks requires innovative architectures that can capture long-term dependencies, robust datasets that cover diverse scenarios, and training strategies that balance visual and language understanding.

This work introduces Valley, a multi-modal foundation model that integrates ViT-L/14 with three distinct temporal modules—average pooling, weighted linear, and transformer—to enhance temporal feature extraction. The two-stage training approach first pre-trains the projection layer for visual-language alignment, then jointly fine-tunes with the LLM, ensuring robust instruction following. Large, high-quality datasets are constructed by filtering WebVid2M and JukinMedia videos, covering diverse tasks like question answering, detailed description, and reasoning. The model’s architecture allows flexible input of varying video lengths and supports multi-task learning, significantly outperforming prior methods in zero-shot and few-shot scenarios. This comprehensive approach addresses previous limitations in data quality, temporal modeling, and multi-task generalization, setting new standards for video multimodal understanding.

• �� Visual encoding: Use ViT-L/14 to extract spatial features from sampled video frames.
• �� Temporal aggregation: Apply three strategies—average pooling, learnable linear weighting, and transformer-based encoding—to fuse temporal information.
• �� Feature fusion: Concatenate spatial, temporal, and global features into a unified representation.
• �� Projection layer: Train a learnable matrix to map visual features into language embedding space.
• �� Two-stage training: Pre-train projection layer with image/video-text pairs; then jointly fine-tune projection layer and LLM.
• �� Data collection: Curate 702k video-text pairs filtered for quality; generate instruction datasets with ChatGPT, covering detailed descriptions, conversations, and reasoning.
• �� Multi-task training: Optimize across diverse tasks—question answering, captioning, reasoning—using extensive instruction data.
• �� Evaluation: Validate on multiple benchmarks, including MSVD, MSRVTT, ActivityNet, and Video-Bench, assessing zero-shot and few-shot performance.

The experimental setup involves training on large-scale datasets like WebVid2M and JukinMedia, with hyperparameters such as a 2×10^-3 learning rate for pre-training and 2×10^-5 for fine-tuning. The model is evaluated on standard benchmarks: MSVD, MSRVTT, and ActivityNet for question answering, with accuracy and ChatGPT-rated scores. Ablation studies compare the three temporal modules, demonstrating that transformer-based modeling yields the best results. Additional tests include video description quality, image understanding on MemeCap, and reasoning on ScienceQA. The experiments utilize 8 Nvidia A100 GPUs, with training epochs carefully tuned to balance performance and resource constraints. The results consistently show that Valley outperforms baseline models, especially in long video comprehension and complex reasoning tasks, validating the effectiveness of the proposed architecture and training strategy.