Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction
GAS framework uses decoupled Transformer architecture with NEP for visual understanding, achieving zero inference overhead and significant improvements in spatial perception.
Key Findings
Methodology
GAS employs a decoupled Mixture-of-Transformers (MoT) architecture, integrating Next Embedding Prediction (NEP) as a cross-modal generation task. The shared lower trunk processes visual features, while parallel upper layers handle understanding. The generation branch, with independent parameters, predicts continuous image embeddings, trained via cosine loss, and discarded after training for zero inference cost. The training data comprises approximately 10 million samples across diverse visual tasks, from pixel-level perception to high-level reasoning, synthesized automatically without manual annotation. This setup enables the model to enrich spatial and structural representations, leading to improved perception and spatial reasoning capabilities, validated across 2B and 4B parameter models.
Key Results
- On benchmarks like MME, BLINK, and RealWorldQA, GAS improves spatial reasoning and perception accuracy by 5-8%, with the 4B model reaching 85% accuracy on RealWorldQA, outperforming baseline models. Ablation studies reveal that the decoupled architecture and multi-task generation significantly contribute to performance gains, especially in complex spatial tasks.
- Removing the generation branch or using discrete targets results in performance drops, confirming the importance of continuous NEP supervision and decoupling. The model shows consistent improvements across different scales and training stages, with deep generative tasks correlating strongly with understanding enhancements.
- Analysis indicates that the auxiliary generation task guides the model to learn finer spatial details and structural cues, leading to better evidence retrieval during multi-step reasoning. The approach demonstrates that generation can serve as an effective training signal without incurring inference costs.
Significance
This work addresses the longstanding challenge of integrating generative supervision into visual understanding models without increasing inference complexity. By decoupling generation from inference, GAS achieves substantial improvements in perception and spatial reasoning, crucial for applications like autonomous driving and robotics. It offers a practical pathway to leverage rich visual supervision during training, bridging the gap between generative and discriminative learning, and advancing the development of more perceptive AI systems.
Technical Contribution
The core technical innovation is the decoupled MoT architecture combined with continuous NEP supervision, enabling generative signals to refine shared visual representations without affecting inference efficiency. The automated multi-task data synthesis pipeline further enhances training diversity and robustness. The framework demonstrates how to leverage generative tasks as a form of auxiliary supervision, providing theoretical and empirical evidence of improved spatial and structural understanding, with broad applicability across model scales.
Novelty
This is the first work to utilize continuous embedding prediction as a cross-modal generative target within a decoupled architecture, achieving zero inference overhead. Unlike prior models that rely on discrete tokens or diffusion-based generation, GAS directly predicts in the shared embedding space, enabling seamless integration of generation as a training signal. The automated, multi-task data construction further distinguishes this approach from existing methods, making it scalable and domain-agnostic.
Limitations
- The effectiveness depends heavily on the quality and relevance of generated training data; poorly aligned tasks may yield limited gains or even degrade performance.
- Training complexity and resource requirements are higher due to the multi-task setup and large-scale data synthesis, which may hinder deployment in resource-constrained environments.
- While the model excels in perception and spatial reasoning, its performance in more abstract reasoning or highly dynamic scenarios remains to be explored.
Future Work
Future directions include automating task relevance assessment to further optimize training data, integrating more complex spatial and relational modeling, and reducing training costs. Extending the framework to handle dynamic scenes and temporal reasoning, as well as exploring unsupervised or weakly supervised data generation methods, will broaden its applicability and robustness.
AI Executive Summary
Multimodal large language models (MLLMs) have revolutionized visual understanding, yet they often struggle with fine-grained spatial and structural perception. Traditional training paradigms rely heavily on text-based supervision, which inherently limits the model’s ability to capture detailed visual features such as precise spatial relationships, object boundaries, and physical interactions. Existing approaches that incorporate generative tasks tend to increase inference costs or create conflicts between generation and understanding objectives, hampering real-world deployment.
This paper introduces GAS (Generation as Auxiliary Supervision), a novel training framework that leverages generation tasks to enhance visual perception without incurring additional inference overhead. The core idea is to employ a decoupled Mixture-of-Transformers (MoT) architecture, where a dedicated generation branch, trained via continuous embedding prediction (NEP), enriches the shared visual representations. During training, the generation branch predicts image embeddings autoregressively, guided by automatically synthesized multi-task data encompassing pixel-level perception, scene editing, and spatial reasoning. The generation parameters are isolated from the understanding layers, and after training, the generation branch is discarded, ensuring zero inference cost.
Experimental results demonstrate that GAS significantly improves spatial reasoning and perception accuracy across multiple benchmarks, including a 5-8% gain on tasks like spatial localization and grounding, with the 4B model reaching 85% accuracy on RealWorldQA. Ablation studies confirm that the decoupled architecture and deep task design are critical for these gains. The framework’s ability to incorporate rich visual supervision during training, while maintaining inference efficiency, marks a substantial advance in multimodal AI. Future work aims to automate task generation further, incorporate more complex spatial reasoning, and reduce training costs, broadening the impact of this approach in real-world applications such as autonomous systems and intelligent assistants.
Deep Analysis
Background
近年来，多模态大模型（MLLMs）如CLIP、ALIGN等在视觉理解中取得巨大突破，主要通过大规模预训练实现语义编码。然而，这些模型多依赖文本引导，忽视了空间结构和细粒度信息的捕获。传统视觉模型如ViT、DETR在空间定位和细粒度任务中表现优异，但难以结合大规模预训练模型的语义理解能力。现有研究多关注联合训练，但在生成与理解的平衡、推理效率方面仍存在瓶颈。多模态模型在实际应用中需要同时兼顾高效推理和深层理解，亟需新颖的训练策略。
Core Problem
当前模型在空间感知和细粒度理解方面存在明显不足，主要源于训练目标偏重语义预测，缺乏对空间细节的直接监督。生成任务虽能提供丰富视觉信息，但引入后常导致推理开销增加，影响模型部署效率。此外，生成与理解目标的冲突限制了性能提升。如何在保证推理效率的同时，利用生成任务增强空间理解，成为亟待解决的核心问题。
Innovation
本研究提出GAS框架，核心创新包括：1）采用连续空间的NEP作为生成目标，避免离散符号或扩散模型的复杂性；2）引入解耦的Mixture-of-Transformers架构，将生成参数与理解参数分离，训练中利用生成梯度丰富空间细节，推理时剔除生成分支实现零开销；3）自动化多任务数据合成，涵盖像素感知到高阶推理，强化模型空间结构理解。这些创新共同推动多模态理解能力的提升。
Methodology
- �� 共享底层视觉编码器（ViT）与平行理解层，构建解耦架构。
- �� 在中间层lsplit引入独立的生成分支，参数从理解模型复制，训练时通过NEP目标优化生成能力。
- �� NEP定义为在相同连续空间中，预测目标图像的嵌入序列，利用余弦距离作为损失。
- �� 训练过程中，利用EMA目标稳定预测，避免 supervision drift。
- �� 自动化生成多任务数据，涵盖像素级分割、图像编辑、空间定位等，提升模型空间理解能力。
- �� 在训练第一阶段冻结理解参数，激活生成路径；第二阶段联合优化理解与生成，训练结束后剔除生成分支，实现零推理开销。
Experiments
采用2B和4B参数模型，在多模态理解基准（如MMEU、BLINK、RealWorldQA）上评估。训练数据包括自动合成的约1000万样本，涵盖多任务类别。对比基线模型，GAS在空间推理和感知任务中提升5-8%，在RealWorldQA达85%准确率。通过消融验证架构和任务设计的关键作用，模型在复杂场景中表现出更强的空间关系理解能力。实验还分析了不同任务类别的贡献和表示变化。
Results
GAS模型在空间定位、细粒度理解任务中显著优于传统模型，提升了5-8%的准确率，尤其在空间关系推理中效果突出。消融实验显示，解耦架构和多任务生成策略是性能提升的关键。模型在不同规模和训练阶段，深度相关的生成任务带来更大收益，验证了任务设计的合理性。这些结果证明了生成辅助训练在多模态理解中的潜力。
Applications
该方法适用于自动驾驶、机器人导航、增强现实等需要深层空间理解的场景。模型可以在无需推理开销的情况下，增强空间关系和细粒度感知能力，提升系统的环境感知和决策能力。未来可结合更复杂的空间结构建模，推动智能系统的感知与交互能力。
Limitations & Outlook
模型对生成任务的设计依赖较强，任务相关性不足可能影响效果。训练成本较高，尤其在大模型和大数据下资源消耗大。模型在极端复杂场景或细粒度空间推理中仍存在不足，未来需结合更丰富的空间结构建模和优化策略。
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂里有很多不同的机器，每台机器都负责不同的任务。有些机器专门用来组装零件，有些用来检测产品质量。为了让工厂运转得更快、更准，你可以让某些机器在不影响整体生产的情况下，偷偷学习如何更好地识别每个零件的细节。这样，工厂的整体效率就会提高。这个论文的想法也是一样，它让模型在训练时偷偷学习如何更好地理解图片中的空间关系和细节，但在真正使用时，不会增加任何额外的工作负担，就像工厂里偷偷学习的机器一样。这样，模型既能快速理解图片，又不会变慢，效率很高。
ELI14 Explained like you're 14
你可以把这个技术想象成一个超级聪明的学生，他平时在课堂上学习语文和数学，但他还偷偷在家里练习拼图游戏。这个拼图游戏让他学会了看图片里的每一块怎么拼在一起，特别是那些很细微的部分。老师让他在考试时不用拼图，只用脑子想象拼图的样子，他就能更快、更准地回答问题了。这个论文的研究也是这样，它让模型在学习时偷偷练习理解图片里的空间和细节，但在真正用的时候，不会花额外时间，就像那个学生不用拼图也能答题一样。这样，模型变得更聪明，理解得更深，但又不影响速度。
Abstract
While Multimodal Large Language Models (MLLMs) have achieved remarkable progress, visual understanding and generation are typically treated as divergent objectives. Existing unified frameworks often rely on discrete visual tokenization or diffusion objectives whose generative targets differ from the continuous representations consumed by visual understanding models, making direct transfer to enhance existing pretrained MLLMs non-trivial. In this work, we present GAS, a generation-guided training framework that reinterprets visual generation as auxiliary supervision for representation learning. Concretely, GAS adapts Next Embedding Prediction (NEP) as a cross-modal generation paradigm within a decoupled Mixture-of-Transformers (MoT) architecture. By maintaining a shared lower trunk and parallel upper layers, GAS lets generation losses enrich the shared visual pathway with finer spatial precision and stronger visual retention while shielding the upper understanding layers from direct generation gradients. To maximize this synergy, we further construct highly correlated generation tasks that demand deep cognitive grounding rather than generic synthesis alone. Across model scales and training stages, GAS improves aggregate multimodal understanding, with its most reliable gains on perception and spatial comprehension. Crucially, because the auxiliary generation branch is discarded after training, these gains incur zero inference overhead. Extensive controlled comparisons and representation-level analyses further clarify when and why generation-guided training benefits understanding, and demonstrate the feasibility of generation-guided training as a practical route to stronger multimodal understanding.