MOSS-VL Technical Report
MOSS-VL is an open-source vision-language model supporting real-time interaction via gated cross-attention, with 11.3B parameters, excelling in proactive response tasks.
Key Findings
Methodology
The model employs a 27-layer vision encoder initialized from Qwen3-VL and a 48-layer language decoder with XRoPE positional encoding. Gated cross-attention layers, inserted every four layers, enable visual information to influence generation without entering the output sequence. Training involves four pre-training stages—vision-language alignment, large-scale multimodal pretraining, high-quality data, and long-context training—and two supervised fine-tuning stages: standard instruction tuning and Realtime-SFT. Synthetic interaction data supervise response timing, silence, and revision behaviors. The architecture maintains visual tokens outside the decoding sequence, enhancing inference speed and response timeliness.
Key Results
- Offline, MOSS-VL-Instruct outperforms comparable models on temporal reasoning datasets (Minerva, TOMATO, VideoMME-Logical).
- In streaming benchmarks, MOSS-VL-Realtime achieves an average score of 66.0 across four tests, surpassing baselines like JoyAI-VL-Interaction and VideoChat.
- The time-to-first-token (TFF) advantage increases from 2.8× to 5.1× with growing visual context, demonstrating efficient visual information utilization.
Significance
This work advances the state-of-the-art in real-time multimodal understanding, enabling AI systems to perceive and respond simultaneously. It addresses longstanding challenges in synchronizing perception and generation, crucial for natural human-AI interactions, and lays a foundation for intelligent assistants and interactive robots capable of proactive, timely responses.
Technical Contribution
The introduction of gated cross-attention isolates visual tokens from the output sequence, combined with XRoPE multi-axis position encoding for precise temporal alignment. The training pipeline synthesizes interaction behaviors, enabling the model to learn when to speak, stay silent, or revise responses dynamically. The architecture supports long sequences and real-time inference, with visual tokens kept outside the decoder, improving efficiency.
Novelty
This is the first model to integrate gated cross-attention with multi-axis positional encoding for synchronized perception and generation in a vision-language setting. Its ability to perform proactive, real-time responses surpasses previous models limited to offline or delayed responses, marking a significant innovation in multimodal AI.
Limitations
- The model's performance may degrade in highly dynamic or occluded scenes, where perception is challenged. Its high parameter count demands substantial computational resources, limiting deployment on edge devices. Synthetic training data, while extensive, may not fully capture real-world variability, affecting generalization. Further research is needed to improve robustness and efficiency.
Future Work
Future directions include optimizing model architecture for lower computational costs, enhancing robustness in complex environments, and integrating reinforcement learning to refine response strategies. Expanding training data with more real-world scenarios will improve generalization, aiming to deploy more versatile and efficient real-time multimodal systems.
AI Executive Summary
In recent years, vision-language models have achieved remarkable success in understanding static images and videos offline. However, real-time interaction remains a significant challenge, as existing models often suffer from latency and lack continuous perception during generation. Addressing this gap, the OpenMOSS team developed MOSS-VL, a pioneering open-source model capable of perceiving while speaking—a true fifth-level (L5) capability in video understanding.
MOSS-VL's architecture integrates a 27-layer vision encoder with a 48-layer language decoder, employing gated cross-attention layers to enable seamless perception during response generation. The model is trained through a comprehensive curriculum, including four pre-training stages—vision-language alignment, large-scale multimodal pretraining, high-quality data, and long-context training—and two supervised fine-tuning stages. A key innovation is the synthesis of interaction data that teaches the model when to speak, stay silent, or revise responses, effectively simulating real-time scenarios.
Experimental results demonstrate the model's superior performance across multiple benchmarks. Offline, it excels in temporal reasoning tasks, outperforming comparable models on datasets like Minerva and TOMATO. In streaming scenarios, MOSS-VL-Realtime achieves an average score of 66.0 across four benchmarks, notably surpassing baselines such as JoyAI-VL-Interaction. Its architecture allows visual tokens to remain outside the decoded sequence, significantly reducing inference latency and increasing the time-to-first-token advantage from 2.8× to 5.1× as visual context grows.
This work marks a substantial step toward natural, proactive human-AI interaction, with broad implications for intelligent assistants, autonomous robots, and real-time surveillance. Despite its impressive capabilities, challenges remain in deploying on resource-constrained devices and ensuring robustness in complex environments. Future efforts will focus on optimizing efficiency, expanding real-world training data, and integrating reinforcement learning to further enhance responsiveness and reliability. Overall, MOSS-VL sets a new standard for synchronized perception and generation in multimodal AI systems.
Deep Analysis
Background
多模态视觉-语言模型经历了从静态理解到动态交互的演变。早期模型如VisualBERT、LXMERT主要解决静态图像理解，随后VideoBERT、VideoLLaMA引入视频理解能力。现有模型多在离线场景中表现优异，但缺乏持续感知与生成同步能力，限制了在动态交互中的应用。近年来，流媒体模型如ALBEF、Florence尝试实现连续输入，但仍未解决感知与生成的同步问题。随着智能助手和交互机器人的需求增长，实时感知成为研究焦点。
Core Problem
核心难题在于如何在生成过程中持续感知视觉信息，实现响应的及时性和准确性。传统模型在生成时会“盲”着视觉输入，导致响应滞后或信息遗漏。实现感知与生成同步，要求模型在保持高效推理的同时，能动态感知场景变化。这不仅涉及架构创新，还需训练策略优化，尤其在复杂、多变场景中保持鲁棒性。
Innovation
本研究提出门控交叉注意力机制，有效隔离视觉信息，确保视觉感知在生成过程中持续进行。结合XRoPE多轴位置编码，模型能准确捕捉多模态信息的时间关系。训练中引入合成交互语料，模拟实时场景，学习何时发言、何时沉默、何时修正。这一架构突破了传统模型的限制，实现感知与生成同步，显著提升主动响应能力。
Methodology
- �� 视觉编码器：27层Transformer，处理高分辨率图像/视频，提取多层特征。• 文本解码器：48层，结合自注意力和门控交叉注意力，确保视觉信息不进入解码序列。• 门控交叉注意力：在每四层中引入，利用tanh门控机制隔离视觉信息。• XRoPE位置编码：在跨注意力中引入三轴（时间、行、列）位置编码，明确多模态时间关系。• 训练策略：四阶段预训练（对齐、多模态预训练、高质量微调、长序列训练），以及指令微调和实时交互微调。• 合成交互语料：模拟实时场景，学习响应时序，包括沉默、发言、修正。• 视觉-文本同步：通过绝对时间戳和缓存机制，实现视觉信息的持续感知。
Experiments
模型在离线场景中通过多种时间推理视频集（Minerva、TOMATO、VideoMME-Logical）验证，优于对比模型。在流媒体基准测试中，模型在四项中三项获得最高平均分（66.0），在主动响应任务中表现优异。参数效率方面，视觉token未进入解码序列，推理速度提升显著。Ablation研究显示门控机制和XRoPE位置编码对性能提升关键。模型在不同视觉上下文长度下表现稳定，验证其强大感知能力。
Results
模型在离线任务中表现优异，领先多个开源模型。在流媒体任务中，平均得分达66.0，明显优于基线37.5。在视觉上下文增加时，时间-首词优势由2.8倍提升至5.1倍，显示出极高的感知效率。参数规模虽大，但推理延迟控制良好，验证了架构设计的有效性。这些结果表明模型在主动响应和时间敏感任务中具有巨大潜力。
Applications
该模型可广泛应用于智能助手、交互机器人、视频监控等场景，支持实时场景理解和主动响应。只需配备标准GPU硬件，即可实现高效推理。未来还可结合强化学习优化响应策略，提升交互自然度，推动多模态智能系统的商业化落地。
Limitations & Outlook
模型在极端复杂场景中的表现仍有限，尤其在快速变化或遮挡严重的环境中可能出现误判。高参数规模带来较大硬件成本，实时推理对硬件要求较高。此外，合成语料虽丰富，但在真实场景中的泛化能力仍需验证，未来需引入更多真实数据进行微调。
Plain Language Accessible to non-experts
想象你在一个厨房里做饭。厨师（模型）需要同时看着食材（视觉信息）和听着指令（语言），在烹饪过程中不断调整。传统厨师只看完所有食材后才开始做菜（离线模型），而你希望厨师能一边看食材一边做饭，随时根据变化调整（实时交互）。MOSS-VL就像这样一个厨师，能同时“看”和“说”，不断感知厨房的变化，及时回应不同的需求。这种能力让机器人或智能助手能更自然、更灵活地与人互动，像个真正懂事的厨师一样。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的拼图游戏，你需要不停地看着拼图块（视觉信息），同时告诉朋友你拼到哪了（语言输出）。普通的机器人只能看完全部拼图后再说话（离线模型），但你希望它能一边看一边说，甚至在拼图还在变的时候就开始描述。MOSS-VL就像这样一个聪明的伙伴，它能在拼图过程中不断观察，随时调整自己的描述，变得更聪明、更贴心。它用一种特别的“门控”方法，确保每次说话都基于最新的拼图状态，就像你在游戏中不断观察和调整一样。这让它在和人交流时变得更自然、更及时。
Abstract
We present MOSS-VL, an open vision-language model family that treats real-time interaction -- perceiving while it speaks -- as a first-class capability. It is co-designed across the stack: the language decoder attends to vision only through gated cross-attention, so the model can naturally see incoming frames while generating; a synthesized interaction corpus supervises when to speak, when to stay silent, and when to revise; and a staged curriculum concentrates all real-time-specific training in one light final stage over a strong offline foundation. Offline, MOSS-VL-Instruct is competitive at comparable scale and leads temporal-reasoning video sets. Across four streaming benchmarks, MOSS-VL-Realtime posts the best average on three (second on the fourth) among open-source streaming models, sweeping the three subsets that squarely test proactive behavior -- 66.0 vs. 37.5 for the best baseline on OmniMMI Proactive Alerting. With 11.3B parameters but visual tokens outside the decoded sequence, MOSS-VL widens its time-to-first-token advantage over same-backbone Qwen3-VL-8B from 2.8x to 5.1x as visual context grows. We release all five checkpoints, the training curriculum, and the real-time inference code at https://github.com/OpenMOSS/MOSS-VL.