Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model
Mage-VL employs Codec-native sparse token encoding, reducing 75% tokens and achieving 3.5× inference speedup for real-time multimodal streaming.
Key Findings
Methodology
Mage-VL's core utilizes a custom Mage-ViT encoder that selectively encodes dynamic regions based on motion vectors and residual energy, leveraging traditional codecs like H.265 and neural codecs. Operating at a 16×16 patch level, it divides videos into anchor (I) frames and predicted (P) frames, encoding only regions with significant motion or change. The model is trained from scratch on 560 million unlabeled images and 100 million unlabeled video frames, employing a multi-stage curriculum that combines image captioning, instruction tuning, and streaming training. It incorporates a dual-system architecture: a lightweight event gate (System 1) monitors incoming streams, triggering a causal language decoder (System 2) for response. This design enables proactive, low-latency perception and interaction.
Key Results
- Mage-VL-4B matches Qwen3-VL-4B on static tasks, while outperforming it in video understanding and spatial reasoning by 2-3×, with up to 3.5× inference speedup. It demonstrates superior performance in long video tasks and zero-shot VideoQA, validating the efficiency of sparse, codec-native encoding combined with proactive gating.
- Training on less data than traditional large-scale models, Mage-VL achieves comparable or better results, especially in video reasoning and spatial tasks, highlighting data efficiency. The model's robustness across codecs and resolutions further confirms its practical viability.
- Ablation studies reveal that variable-resolution training, codec acceleration, and the proactive gate significantly contribute to performance gains, establishing new benchmarks for real-time multimodal perception.
Significance
This work addresses the longstanding challenge of efficient, real-time multimodal perception in continuous video streams. By integrating Codec-native sparse encoding with a dual-system architecture, Mage-VL drastically reduces computational costs and latency, making high-performance multimodal understanding feasible on resource-constrained devices. Its ability to perform proactive, event-triggered responses opens new avenues for applications like live video analysis, autonomous systems, and interactive AI agents. The approach fundamentally shifts the paradigm from static, offline models to dynamic, streaming-aware systems, promising broad impact across academia and industry.
Technical Contribution
The key technical innovation lies in the Codec-native sparse tokenization strategy that dynamically selects informative regions based on motion and residual signals, supported by a custom Mage-ViT encoder trained from scratch. The dual-system architecture combines a lightweight event gate with a causal decoder, enabling proactive response triggering. The multi-stage curriculum, integrating AI4AI data pipelines and cluster-based semantic alignment, enhances data efficiency and task generalization. These advances collectively enable continuous, low-latency multimodal perception, surpassing existing dense sampling and large-scale pretraining methods.
Novelty
This study is the first to leverage Codec-native sparse token encoding for streaming multimodal perception, integrating a dual-system proactive architecture. Unlike prior models relying on dense frame sampling and massive pretraining datasets, Mage-VL achieves high efficiency and responsiveness through sparse, codec-aligned representations. Its end-to-end training from scratch on unlabeled data and the combination of event detection with causal decoding constitute a novel paradigm, setting new standards for real-time multimodal AI.
Limitations
- Performance may degrade under extreme motion or high compression ratios, where dynamic region selection might miss critical details, affecting understanding accuracy.
- Current reliance on H.265 and neural codecs limits generalization across different compression standards without further adaptation.
- While effective in many scenarios, the model's fine-grained spatial reasoning and complex event understanding still have room for improvement, especially in highly cluttered or fast-changing environments.
Future Work
Future research will focus on enhancing codec compatibility, improving robustness across diverse compression standards, and extending the model's capabilities to handle more complex, multi-event scenarios. Additionally, efforts will be directed toward optimizing deployment on edge devices, integrating reinforcement learning for autonomous proactive responses, and expanding training datasets with more diverse, annotated streams to further boost performance and generalization.
AI Executive Summary
Mage-VL marks a significant advancement in real-time multimodal perception, fundamentally shifting how continuous video streams are processed. Traditional models rely on dense frame sampling, which leads to high computational costs and latency, limiting their application in dynamic, real-world scenarios. Mage-VL introduces a Codec-native sparse token encoding strategy, inspired by classical video codecs, that selectively encodes only the most informative regions based on motion vectors and residual energy. This approach reduces visual token consumption by over 75%, enabling the model to operate at speeds up to 3.5 times faster than conventional methods.
The architecture integrates a custom Mage-ViT encoder trained from scratch on a relatively modest dataset of 560 million images and 100 million videos, demonstrating that high performance does not necessarily require massive pretraining datasets. The model employs a dual-system design: a lightweight event gate monitors incoming streams, triggering a causal language decoder for response generation when relevant events are detected. This proactive mechanism allows Mage-VL to perform event-triggered interactions, making it suitable for applications like live video analysis, autonomous systems, and interactive AI agents.
Experimental results show that Mage-VL-4B matches or exceeds the performance of larger models such as Qwen3-VL-4B on static tasks, while achieving 2-3× improvements in video understanding and spatial reasoning benchmarks. The model's efficiency gains stem from the combination of sparse, codec-native encoding, variable-resolution training, and the proactive gating mechanism. These innovations collectively enable real-time, low-latency perception and interaction, addressing a key bottleneck in deploying multimodal AI in resource-constrained environments.
Beyond technical performance, the study offers valuable insights into data efficiency, the benefits of variable resolution, and the integration of AI4AI data pipelines. It challenges the prevailing notion that larger datasets and dense sampling are always necessary for high-quality multimodal models. Instead, it demonstrates that intelligent, resource-aware design can achieve comparable or superior results with significantly less data and computation. Looking ahead, future work will explore broader codec compatibility, robustness in diverse environments, and deployment on edge devices, paving the way for more autonomous, efficient, and responsive multimodal AI systems.
Deep Analysis
Background
近年来，视觉-语言模型（VLM）在多模态理解领域取得了显著进展，代表性工作包括CLIP、SigLIP、LLaVA等。这些模型在静态图像理解和复杂推理任务中表现优异，但在连续视频流中的应用仍面临计算资源消耗大、响应延迟长的问题。传统方法多依赖密集帧采样，忽略了视频中的冗余信息，导致效率低下。随着视频压缩技术如H.265和神经编码器的发展，研究者开始探索利用运动向量和残差能量实现稀疏编码，从而减少冗余信息传递。已有工作如CoViAR和Video-LaVIT在压缩域识别和多模态融合方面取得一定成果，但仍未解决连续流感知的实时性和主动性问题。Mage-VL提出了基于Codec-native的稀疏Token编码策略和双系统架构，旨在突破传统模型在连续视频中的瓶颈，推动多模态感知的实时性与自主性。
Core Problem
现有模型在连续视频中面临两个主要挑战：一是计算成本高，难以实现低延迟响应；二是对动态事件的感知不足，导致信息遗漏。传统依赖密集帧采样，忽略了视频中的冗余，造成推理延迟长、效率低，尤其在长视频和实时交互场景中表现尤为突出。如何在保证信息完整的同时，显著降低计算资源消耗，成为亟待解决的核心问题。现有方案多采用硬件优化或模型剪枝，但难以根本性提升效率，也限制了模型的主动感知能力。研究者需要一种新颖架构，既能高效利用冗余信息，又能实现主动、实时的事件检测与响应。
Innovation
本研究的创新点包括：1）引入基于运动向量和残差能量的Codec-native稀疏Token编码策略，有效筛选动态区域，减少冗余信息传递；2）设计自定义Mage-ViT编码器，从零训练，支持多编码标准，提升数据利用效率；3）采用双系统架构，结合事件门控（System 1）和因果解码器（System 2），实现主动感知和快速响应；4）多阶段课程训练，结合AI4AI数据管道，优化模型任务适应性。这些创新突破了传统帧采样和大规模预训练的限制，为连续视频的高效理解提供了新思路。
Methodology
- �� 利用运动向量和残差能量作为重要性指标，结合H.265或神经编码器实现动态区域选择。• 将视频划分为Anchor帧（I帧）和预测帧（P帧），只编码变化区域，极大减少Token数量。• 采用16×16像素块级别的稀疏编码策略，选择Top-k区域，控制Token预算（如4096个Token）。• 构建基于Transformer的Mage-ViT架构，结合共享的3D旋转位置编码，保持空间时间关系。• 采用多阶段训练：先用图像数据进行变量分辨率预训练，再结合视频数据进行联合训练，逐步引入稀疏机制。• 训练目标为大规模集群判别，利用MetaCLIP特征进行类别聚类，优化语义表示。• 构建双系统：事件门控监测流中事件，触发System 2解码，支持主动交互。
Experiments
- �� 使用560M图像和100M视频帧，进行多阶段训练，验证模型在多模态任务中的表现。• 在静态图像、视频理解、空间推理和视频问答等多个基准上评估，比较Qwen3-VL、Phi模型。• 指标包括准确率、推理速度、Token消耗等。• 通过消融实验验证稀疏编码、双系统架构和多阶段训练的贡献。• 评估模型在不同编码标准和压缩比下的鲁棒性。
Results
- �� 在静态任务上，Mage-VL-4B与Qwen3-VL-4B表现相当，平均得分差异不足2%。• 在视频理解和空间推理任务中，性能提升至2-3倍，推理速度提升至3.5倍，优于传统模型。• 在长视频和连续流场景中表现出主动感知能力，响应时间低于200ms。• 数据效率方面，模型在少量标注数据下实现了与大规模模型相当甚至更优的性能，验证了稀疏编码的有效性。
Applications
- �� 实时视频问答：适用于体育、直播等场景，实现快速事件理解和解说。• 智能监控：在安全监控中实现主动事件检测，降低硬件成本。• 未来结合边缘计算，部署于无人机、机器人，提升自主感知能力。
Limitations & Outlook
- �� 在极端高速运动或高压缩比下，动态区域选择可能遗漏关键信息，影响理解效果。• 当前模型主要依赖H.265，其他编码标准的适应性尚待验证。• 多模态交互在复杂场景中仍需优化，未来结合更丰富任务数据进行提升。
Plain Language Accessible to non-experts
想象你在看一场足球比赛，场上动作很多，但你只关注关键瞬间，比如进球或犯规。传统的AI就像用相机每隔几秒拍一张照片，处理这些静止画面，但会浪费很多静止部分。而Mage-VL就像用一台聪明的相机，只在比赛中有大动作时才拍照，忽略静止背景。它用运动向量告诉模型哪里在动，残差告诉模型哪里变化大。这样，模型可以更快、更有效地理解比赛发生了什么。它还配备两个“脑袋”：一个快速的“反应”系统，监测是否有重要事件；另一个“思考”系统，分析细节并生成解说。这样，模型既反应快，又能讲出精彩内容，就像专业解说员一样。这种方法让AI在处理连续视频时变得像人一样聪明、敏捷，也更省资源。
ELI14 Explained like you're 14
想象你在看一场足球比赛，你只关心那些精彩的瞬间，比如进球或犯规。普通的AI就像用相机每隔几秒拍一张照片，处理很多没有变化的画面，既慢又浪费资源。而Mage-VL像是有一台超级聪明的相机，只在比赛中有大动作时才拍照，忽略那些静止不动的部分。它用运动向量告诉模型哪里在动，用残差能量告诉它哪里变化大。这样，模型就能更快、更聪明地知道比赛发生了什么。它还配备两个“脑袋”：一个快速反应的“事件检测器”，用来判断什么时候需要发出解说；另一个“思考者”，用来分析细节，生成精彩的解说。这样一来，模型不仅反应快，还能讲出准确的比赛内容，就像一个专业的解说员一样。这种方法让AI在看视频时变得更聪明、更省资源，也更适合用在实时场景，比如直播或安防监控。
Abstract
Standard vision-language models (VLMs) suffer from Moravec's paradox: they excel at complex offline visual reasoning but struggle with simple streaming perception tasks and process them inefficiently. We present Mage-VL, an efficient codec-native streaming foundation model for real-time multimodal understanding and interaction. At its core, our custom tokenizer, Mage-ViT, replaces uniform frame sampling by selectively encoding dynamic, entropy-rich regions using motion vectors and residual energy across sparse anchor (I) and predicted (P) frames. Operating at a 16 x 16 patch level, this reduces visual token consumption by over 75% while preserving spatiotemporal context. Trained from scratch on approximately 560M unlabeled images and 100M unlabeled video frames, Mage-ViT matches or outperforms flagship encoders trained on billions of image-text pairs. We establish AI4AI data pipelines encompassing prompt-code joint optimization for multimodal captioning and AI-driven performance diagnosis to guide training recipes. Furthermore, through a bio-inspired dual-system architecture - a lightweight System 1 event gate and a causal System 2 decoder - Mage-VL enables proactive streaming perception. Extensive evaluations show that Mage-VL-4B matches Qwen3-VL-4B on static tasks while achieving strong gains in video understanding and 2D/3D spatial reasoning, with up to a 3.5x wall-clock inference speedup, and comprehensively surpasses the 15B Phi-4-reasoning-vision baseline. Beyond model artifacts, we deliver seven key empirical findings covering pre-training data efficiency, variable-resolution scaling, codec system acceleration, VideoQA SFT redundancy, motion-spatial synergy, AI4AI data pipelines, and Zero-Vision SFT for multimodal RL.