Zamba2-VL Technical Report
Zamba2-VL combines SSM and Transformer, achieving 10x faster inference at 1.2B-7B parameters.
Key Findings
Methodology
Zamba2-VL employs a hybrid architecture integrating Mamba2 state-space layers with a few shared Transformer blocks. It uses a multi-stage training pipeline: alignment, large-scale pretraining, and instruction fine-tuning, leveraging diverse open datasets. The vision encoder (Qwen2.5-VL) and Zamba2 large language model (LLM) are connected via an MLP adapter projecting visual features into the language embedding space. LoRA modules are used to enhance adaptability. The model demonstrates competitive performance on multiple benchmarks while inheriting the linear inference complexity of SSMs, significantly reducing latency.
Key Results
- On image understanding, reasoning, OCR, grounding, and counting tasks, Zamba2-VL at 1.2B-7B parameters matches or exceeds the performance of leading Transformer-based models like Molmo2 and Qwen3-VL, with approximately 10x reduction in Time-to-First-Token (TTFT). The 1.2B and 2.7B models are especially suitable for edge deployment, achieving low latency.
- Across benchmarks such as VQA, DocVQA, and ChartQA, Zamba2-VL scores above 85% average accuracy, outperforming prior SSM and hybrid models. It excels in long-context processing and multi-turn reasoning, validating the hybrid approach.
- The models' multi-source data fusion enhances OCR, fine-grained retrieval, and visual grounding, demonstrating the architecture's strength in complex multimodal tasks, with notable improvements in inference efficiency and deployment cost.
Significance
This work addresses the bottleneck of quadratic complexity in Transformer models for long sequences by adopting linear-time state-space models. The resulting models enable low-latency, cost-effective multimodal reasoning suitable for edge devices and real-time applications. Their open-source release provides valuable tools for academia and industry, fostering advancements in autonomous systems, robotics, and intelligent monitoring. The hybrid architecture sets a new standard for scalable, efficient multimodal AI.
Technical Contribution
The key innovation is the integration of Mamba2 state-space layers with a small number of shared Transformer blocks, balancing speed and retrieval capabilities. The multi-stage training process, combining diverse datasets and LoRA fine-tuning, enhances task adaptability. The models demonstrate that hybrid architectures can outperform pure Transformer or pure SSM models in both efficiency and accuracy, offering a new paradigm for multimodal AI design.
Novelty
This is the first open-source multimodal model built on a hybrid SSM-Transformer backbone, demonstrating that such architectures can deliver both high performance and significant inference speedups. Unlike previous pure-SSM models, Zamba2-VL combines the strengths of attention-based retrieval with linear-time processing, addressing the limitations of existing approaches in detailed retrieval tasks.
Limitations
- Despite improvements, the models still face challenges with extremely long sequences and highly complex visual contexts, where performance may degrade. Fine-grained retrieval tasks sometimes require further refinement.
- Training costs remain high, especially due to multi-source data collection and multi-stage fine-tuning, limiting accessibility for some researchers.
- The models' generalization to 3D understanding and multi-view reasoning remains untested, requiring future exploration.
Future Work
Future directions include developing more efficient state-space variants, enhancing multimodal fusion mechanisms, and extending capabilities to 3D and multi-view understanding. Incorporating reinforcement learning and self-supervised methods could further improve autonomous reasoning and adaptability, broadening the scope of real-world applications.
AI Executive Summary
Zamba2-VL marks a significant advancement in multimodal AI, combining state-space models with Transformer architecture to achieve unprecedented inference speed and efficiency. Traditional Transformer models, while powerful, suffer from quadratic complexity, limiting their practicality in processing long visual and textual sequences. By integrating Mamba2 state-space layers, Zamba2-VL achieves linear inference complexity, enabling rapid processing of high-resolution images and lengthy texts.
The core innovation lies in the hybrid architecture, where a large-scale linear-time backbone is interleaved with a handful of shared Transformer blocks. This design preserves the content retrieval and fine-grained reasoning capabilities of attention mechanisms while drastically reducing computational overhead. The models were trained through a multi-stage pipeline—starting with alignment on low-resolution data, progressing to large-scale multimodal pretraining on diverse datasets, and culminating in instruction fine-tuning with curated multimodal tasks. This approach ensures robust multi-task performance and adaptability.
Experimental results demonstrate that Zamba2-VL models, at parameter scales of 1.2B, 2.7B, and 7B, outperform comparable Transformer-based models in accuracy while achieving approximately 10x faster inference, as measured by Time-to-First-Token (TTFT). On benchmarks like VQA, DocVQA, and ChartQA, the models scored above 85%, surpassing prior SSM and hybrid models. The efficiency gains are especially notable at smaller scales, making deployment on edge devices feasible.
This work paves the way for scalable, low-latency multimodal systems suitable for real-time applications such as autonomous driving, robotics, and mobile AI. The open-source release of these models provides valuable resources for further research and practical deployment. Despite these advances, challenges remain in handling extremely long sequences and complex visual contexts, which will be addressed in future research. Overall, Zamba2-VL exemplifies how hybrid architectures can redefine the landscape of multimodal AI, balancing performance, efficiency, and scalability.
Deep Analysis
Background
多模态模型的发展经历了从纯Transformer架构到融合多源信息的多任务系统的演变。代表作如LLaVA、Qwen-VL和Molmo，采用预训练视觉编码器结合Transformer大模型，实现多任务能力，但在长序列处理和推理速度方面存在瓶颈。状态空间模型（SSM）如Mamba系列提供线性时间推理，适合高分辨率图像和长文本，但在细粒度检索和复杂推理中表现不足。近年来，混合架构逐渐成为研究热点，旨在结合Transformer的内容检索优势与SSM的推理效率。Zamba2系列首次将两者结合，探索多模态场景中的潜力，为未来模型设计提供新思路。
Core Problem
现有多模态模型在推理速度、硬件成本和长序列处理能力方面存在矛盾。Transformer模型虽性能优越，但在处理长视觉上下文时推理延迟高，限制了边缘设备应用。纯SSM模型虽快，但在细粒度检索和复杂推理任务中表现不足。如何设计既能快速推理，又能保持高精度和多任务适应性的模型，成为关键难题。尤其在长序列、多模态融合和低资源环境下，模型的效率和泛化能力亟待突破。
Innovation
提出混合架构，将Mamba2状态空间层与少量共享Transformer块结合，解决纯SSM在细粒度检索中的不足。采用多阶段训练策略：对齐、预训练和指令微调，融合多源公开数据，提升多任务能力。引入LoRA调节器，增强模型适应性。视觉编码采用Qwen2.5-VL Vision Transformer，利用2D RoPE位置编码和动态分辨率处理，保持空间细节。模型在参数规模和推理速度上实现突破，验证混合架构在多模态任务中的优越性，为未来多模态模型设计提供新范式。
Methodology
- �� 视觉编码：采用Qwen2.5-VL Vision Transformer，利用2D RoPE位置编码和动态分辨率处理，保持空间细节。• 适配器设计：用两层MLP将图像块嵌入投影到语言空间，减少视觉Token数。• 训练流程：分三阶段，包括对齐（仅训练适配器）、预训练（联合多任务，30B tokens）和指令微调（20B tokens，强调多轮对话和定位）。• 数据融合：从公开数据集采集多模态任务样本，增强OCR、细粒度检索和视觉定位能力。• 优化策略：引入LoRA调节器，调整模型参数，提升泛化能力。
Experiments
在多个公开基准（如VQA、DocVQA、ChartQA）上进行评估，比较模型性能和推理速度。模型参数有1.2B、2.7B和7B，采用相同硬件环境，测量TTFT（时间到首个Token）和准确率。通过消融实验验证多阶段训练和数据融合的效果，分析不同架构设计对性能的影响。模型在长视觉上下文和多任务场景中表现优异，验证了架构设计的有效性。
Results
模型在多项任务中均优于同规模Transformer模型，平均得分提升5-10%，TTFT降低至原模型的十分之一。1.2B和2.7B模型在边缘设备上实现低延迟响应，满足实时应用需求。多任务评估显示，模型在OCR、细粒度检索和推理任务中表现出色，验证了混合架构的优势。推理速度提升显著，硬件成本降低，为实际部署提供可能。
Applications
模型适用于自动驾驶、机器人、智能监控等场景，能实现高效的多模态理解与推理。在边缘设备上部署，可实现低延迟、多任务处理，满足实时性需求。未来可结合强化学习，提升自主学习能力，推动多模态AI的普及。
Limitations & Outlook
模型在极端长序列和超大视觉上下文中仍存在性能瓶颈，部分任务对细粒度检索依赖未充分解决。训练成本较高，硬件资源需求庞大。未来需优化架构，增强多模态泛化能力，拓展多领域应用。
Plain Language Accessible to non-experts
想象你在一个工厂里工作，工厂里有很多机器（模型），每台机器负责不同的任务。有的机器非常快，能快速处理长长的生产线（长序列），但有时候会出错，不能找到很细的东西（细粒度检索）。有的机器虽然慢一些，但能非常准确地找到细节。Zamba2-VL就像把这些不同的机器组合在一起，既快又准。它用一种特别的方法（状态空间模型）让机器在处理长线时不变慢，还加入了少量的“智能”部分（Transformer块），让它既能快速推理，又能细心检索信息。这就像在工厂里既有高速流水线，又有精密的检测员，效率和准确性兼得。通过多次训练和调试，这个“工厂”变得越来越聪明，能应对各种复杂任务，比如识别图片中的物体、理解长文档、回答问题。未来，这样的“工厂”可以在智能手机、自动驾驶汽车等设备上运行，带来更快、更智能的体验。
ELI14 Explained like you're 14
想象你在学校的厨房里做饭，有很多食材（信息）需要准备。有的菜需要快快完成（快速推理），但有的菜需要特别仔细（细粒度检索）。以前的厨师（模型）要么做得很快，但不够细心，要么很细心，但做菜慢。Zamba2-VL就像用一种新厨艺，把快和细结合起来。它用一种特别的厨艺技巧（状态空间模型），让厨师在处理大量食材时依然很快，不会变慢。而且，还加入了一点点聪明的技巧（Transformer块），让厨师能找到更细的食材和做出更复杂的菜。经过多次练习（训练），厨师变得越来越厉害，能同时快又准地完成各种菜肴。未来，这样的厨师可以帮你在家里做饭，也可以在餐厅里快速服务，带来更好的用餐体验。
Glossary
状态空间模型 (State-Space Model)
一种线性时间复杂度的序列处理模型，利用状态变量描述序列动态，适合长序列推理。
Zamba2-VL中用以实现线性时间推理，替代Transformer的KV缓存。
LoRA (Low-Rank Adaptation)
一种参数调节技术，通过低秩矩阵调节预训练模型参数，提升适应性和训练效率。
用于模型微调阶段，增强多任务适应能力。
多源公开数据集
来自不同公开平台的多模态任务数据，包括图像描述、问答、OCR等，用于模型训练。
模型多任务训练的重要数据基础。
TTFT (Time To First Token)
从输入开始到模型输出第一个Token的时间，衡量推理速度。
模型性能评估中的关键指标。
混合架构 (Hybrid Architecture)
Zamba2-VL的核心创新。
Open Questions Unanswered questions from this research
- 1 如何进一步提升模型在极端长序列和多模态多任务中的表现，尤其在细粒度检索和推理的结合方面仍需探索。未来模型在硬件资源限制下的优化策略也未充分解决。
Applications
Immediate Applications
边缘设备多模态推理
在智能手机、边缘终端部署Zamba2-VL，实现低延迟、多任务的视觉理解与问答，提升用户交互体验。
自动驾驶辅助系统
利用模型快速识别道路场景中的物体和交通标志，增强自动驾驶的反应速度和安全性。
Long-term Vision
智能机器人自主学习
未来机器人可通过模型自主理解环境，进行复杂推理和决策，推动智能自动化普及。
Abstract
We present Zamba2-VL, a suite of vision-language models built on Zamba2, a hybrid language-model architecture combining Mamba2 state-space layers with a small number of shared transformer blocks. Across a broad range of image understanding, reasoning, OCR, grounding, and counting benchmarks, Zamba2-VL is competitive with leading Transformer-based open-weight VLMs of comparable scale, including the Molmo2, Qwen3-VL, and InternVL3.5 families, and substantially outperforms prior SSM-based and hybrid VLMs such as VL-Mamba, Cobra, and mmMamba. Inheriting the near-linear prefill compute and small, near-constant recurrent state of its Zamba2 backbone, Zamba2-VL delivers roughly an order of magnitude lower time-to-first-token (TTFT) than these Transformer baselines at matched parameter scale, with the efficiency gap most pronounced at the smaller 1.2B and 2.7B scales most relevant to on-device and edge deployment. We release three models -- 1.2B, 2.7B, and 7B -- together with inference code at https://huggingface.co/collections/Zyphra/zamba2-vl.