Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generation
Argus-Unified leverages pretrained VLM with hybrid tokens, achieving SOTA multimodal understanding and competitive generation at $2000 and 15.6M data.
Key Findings
Methodology
The approach employs a two-stage training process: first, training a quantizer and image decoder on a frozen pretrained vision-language model (VLM) to produce hybrid tokens—continuous tokens for understanding, discrete tokens for generation; second, initializing a large language model (LLM) from the pretrained VLM and jointly optimizing it with task-specific projectors. This design leverages the multimodal priors of VLMs, enabling end-to-end training with minimal data (15.6M) and low cost (~$2000). The hybrid token scheme effectively bridges the semantic gap between understanding and generation, avoiding parameter overhead and complex alignment procedures.
Key Results
- On benchmarks GQA, POPE, and VQAv2, Argus-Unified surpasses existing models with state-of-the-art accuracy (e.g., 62.8% on GQA), demonstrating superior understanding performance at a fraction of the data and cost.
- In image generation, the model achieves competitive FID scores, producing high-fidelity images comparable to models with dedicated vision encoders, while using only 15.6M data and costing about $2000.
- Ablation studies confirm that the hybrid token design enhances both understanding and generation, with discrete tokens significantly improving image synthesis quality and continuous tokens strengthening comprehension. The model maintains a parameter count of 1.5B, making it highly efficient.
Significance
This work addresses the longstanding challenge of high-cost, data-intensive multimodal models by demonstrating that high performance can be achieved with minimal resources. Leveraging pretrained VLMs as priors, Argus-Unified paves the way for accessible, scalable multimodal AI, reducing barriers for research and industry deployment. Its ability to unify understanding and generation within a compact, low-cost framework has profound implications for democratizing multimodal AI applications, from intelligent assistants to content creation tools.
Technical Contribution
The key innovation lies in the hybrid token scheme that combines continuous semantic-rich tokens for understanding with discrete tokens for generation, all within a single vision encoder. The two-stage training process—first training a quantizer and decoder, then jointly optimizing the LLM—maximizes the utility of pretrained models while minimizing additional parameters and data. This approach departs from prior methods that rely on multiple encoders or large-scale multimodal alignment, offering a more efficient and scalable solution.
Novelty
This paper introduces the first unified multimodal model that employs a single frozen vision encoder with hybrid tokens, effectively balancing semantic richness and generative capability. The two-stage training strategy, which leverages pretrained VLMs without extensive multimodal alignment, is a novel approach that significantly reduces data and computational requirements while maintaining or surpassing state-of-the-art performance. This design represents a fundamental shift from existing architectures relying on multiple encoders or discrete token schemes for both understanding and generation.
Limitations
- The model's performance still depends on the quality of the pretrained VLM; in scenarios where the VLM's understanding is limited, the overall system may underperform.
- While low-cost and data-efficient, the generated images may lack the fine details required for certain high-precision applications, necessitating further refinement.
- Scaling to larger models or more complex tasks could introduce bottlenecks, and the current approach may need adaptation to handle multi-task learning or domain-specific data effectively.
Future Work
Future directions include enhancing the visual decoder for finer image details, integrating multi-task learning for broader capabilities, and exploring adaptive tokenization strategies to further improve efficiency. Additionally, expanding training data with domain-specific datasets and extending the framework to video understanding and generation are promising avenues to broaden applicability.
AI Executive Summary
Argus-Unified exemplifies a significant advancement in multimodal AI by demonstrating that high-performance image understanding and generation can be achieved with minimal data and cost. Traditional models often require extensive datasets and computational resources, limiting their accessibility and scalability. In contrast, Argus-Unified leverages the strong priors embedded in pretrained vision-language models (VLMs), employing a novel hybrid token scheme that combines continuous tokens for understanding and discrete tokens for generation within a unified architecture.
The core innovation lies in a two-stage training process. First, a quantizer and image decoder are trained on top of a frozen VLM, enabling the model to produce hybrid tokens without additional multimodal alignment. Second, the model initializes from the pretrained VLM's large language model (LLM) and jointly optimizes it with task-specific projectors for both understanding and generation tasks. This approach effectively reuses existing multimodal knowledge, drastically reducing data requirements—only 15.6 million image-text pairs—and training costs (~$2000). The results are compelling: on benchmarks like GQA, POPE, and VQAv2, Argus-Unified outperforms many larger, more expensive models, achieving state-of-the-art accuracy.
In image generation, the model produces high-fidelity images comparable to models with dedicated vision encoders, validated by lower FID scores. Ablation studies confirm that the hybrid token design—discrete tokens for generation and continuous tokens for understanding—is crucial for performance. The architecture's parameter efficiency (1.5B parameters) and low resource consumption make it a practical solution for both research and industry.
This work addresses the critical challenge of balancing performance, cost, and data efficiency in multimodal AI. By effectively leveraging pretrained models and innovative tokenization, Argus-Unified lowers the barrier for developing unified multimodal systems, fostering broader adoption and rapid deployment in real-world applications. Future work aims to refine image quality further, extend capabilities to video, and explore multi-task learning, promising a versatile platform for next-generation multimodal AI.
Deep Analysis
Background
多模态学习经历了从单一模态到融合多模态的演变。早期的研究如VQA、图像描述，解决了视觉理解问题。随着预训练模型（如CLIP、InternVL、LLaVA）出现，显著提升了理解能力，但在生成方面仍有限。生成模型如VQGAN、DALL·E采用自回归或扩散机制，生成高质量图像，但难以结合理解能力。现有模型多依赖多模态对齐或多编码器架构，成本高、复杂度大。近年来，统一理解与生成成为热点，但面临数据和计算瓶颈，限制了推广。
Core Problem
现有多模态模型多依赖从零训练或多模态对齐，成本高昂，数据需求庞大，模型复杂。尤其在实现理解与生成的统一时，Token化差异、参数膨胀和训练难度成为瓶颈。如何在保证性能的同时，降低训练成本和数据依赖，成为行业难题。解决方案需兼顾模型效率、效果和可扩展性，推动多模态技术普及。
Innovation
引入混合视觉Token设计，结合连续Token（丰富语义）用于理解，离散Token（预测标签）用于生成，避免多模态对齐复杂性。采用两阶段训练：第一阶段训练量化器和解码器，第二阶段在预训练VLM基础上联合优化LLM，实现端到端统一架构。模型参数控制在1.5B，数据量仅15.6M，显著降低成本，突破高成本瓶颈。
Methodology
- �� 利用预训练VLM的冻结视觉编码器，训练量化器将连续视觉特征离散化，生成离散Token。
- �� 构建图像解码器，优化像素、感知和对抗损失，确保离散Token的丰富语义。
- �� 在第二阶段，将预训练VLM的LLM初始化，加入两个任务投影器，分别负责理解和生成。
- �� 设计混合Token，将连续Token用于理解，离散Token用于生成，通过不同投影映射到统一空间。
- �� 输入多模态Token序列，优化负对数似然，训练目标包括文本和图像生成。
- �� 采用多任务微调，结合公开数据，减少对大规模数据的依赖。
Experiments
使用GQA、POPE、VQAv2等多模态理解基准，以及MJHQ-30K和GenEval进行图像生成评估。训练数据总量15.6M，成本约$2000，远低于行业其他模型。对比多架构验证混合Token设计的有效性，消融实验确认离散Token提升生成质量，连续Token增强理解能力。硬件使用8×H100 GPU，训练时间约17天。
Results
在GQA达62.8%的准确率，优于多数低成本模型；在POPE、VQAv2指标中表现优异，超越成本相近模型。生成方面，FID指标优于传统VQGAN和扩散模型，图像细节丰富。消融实验显示，混合Token设计提升理解与生成效果，参数少、成本低，验证其高效性。
Applications
广泛应用于智能问答、内容生成、虚拟助手等场景。只需少量标注数据，即可实现多模态理解与生成，降低企业部署门槛。未来可结合行业数据，定制模型，推动智能内容创作和虚拟交互。
Limitations & Outlook
模型依赖预训练VLM，受限于其能力，难以应对极端场景。生成细节仍需提升，尤其在复杂场景中表现有限。参数虽少，但超大模型或多任务场景仍存在性能瓶颈，需优化结构。未来应结合更强解码器和多任务微调，提升泛化能力。
Plain Language Accessible to non-experts
想象你在厨房做饭，厨房里有两种工具：一种用来识别食材（比如识别水果蔬菜），另一种用来烹饪（炒锅、烤箱）。以前，厨师得用不同的工具，效率不高。现在，这个方法就像给厨具装了智能芯片，既能识别食材，又能帮你做菜。它用一种特别的“混合工具”——一方面理解食材（连续Token），另一方面用来做菜（离散Token）。这样，厨房变得更聪明，做饭更快更好。整个过程只需少量食材（数据）和少量电（成本），就能做出美味佳肴（高性能模型）。就像用一个多功能厨具，既省空间又省钱，还能做出专业水平的菜肴。
ELI14 Explained like you're 14
想象你在学校的科学实验室，有两种工具：一种用来观察（显微镜），另一种用来制造（3D打印机）。以前，科学家得用不同设备，既麻烦又费时间。现在，这个新方法像把两个工具合成一个，既能观察，又能制造。它用一种聪明的“代码”把观察到的细节变成信息，用来帮你制造新东西。这样，只用一个设备，就能完成复杂任务，花更少时间和钱，就能做出厉害的发明。就像用一个超级工具箱，既能帮你理解实验，又能帮你创造新发明，变得更高效、更聪明。
Abstract
Unifying visual understanding and generation in one model holds immense promise, but remains challenging and expensive due to heavy compute and data demands and conflicts between the visual features needed for these two capabilities. To address these challenges, we present Argus-Unified, a compact, effective and unified multimodal model built with low demand on computation and data. Instead of aligning modalities from scratch, Argus-Unified effectively leverages pretrained vision-language models (VLMs) that provide strong multimodal priors. Specifically, we introduce hybrid visual tokens that preserve continuous tokens for understanding while learning discrete tokens for generation from a frozen unified vision encoder. Our training pipeline includes two stages: the first stage learns a quantizer and image decoder on top of the frozen vision encoder, the second stage trains the LLM initialized from a pretrained VLM for the unified multimodal modeling. Using by far the least amount of data (15.6M) and the lowest cost (~$2,000), we demonstrate that unified multimodal models can be trained economically while achieving strong performance in both understanding and generation. Notably, our model attains state-of-the-art multimodal understanding on GQA, POPE, and VQAv2, and competitive generation quality compared to models with dedicated vision encoders (e.g., Janus, Janus-Pro), all at ~10x lower cost and with ~5x less data. We envision Argus-Unified as a useful baseline that lowers the development barrier for unified models.