A Visual Dependence-Aware Framework for Multimodal Unsupervised Continual Post-Training
VDA framework with VC-OT and VMA effectively mitigates visual dependence drift in unsupervised continual multimodal learning, achieving 62.5% accuracy across six tasks.
Key Findings
Methodology
This study introduces VDA, combining Visually Constrained Optimal Transport (VC-OT) and Visually Modulated Adaptation (VMA). VC-OT models the structural drift of visual dependence as an optimal transport problem, utilizing region-aware costs and dependence-stratified penalties to preserve old-task visual structures and prevent global shifts. VMA dynamically adjusts learning weights based on the target token's visual dependence, emphasizing visually grounded knowledge and suppressing language bias. These mechanisms work synergistically to balance stability and plasticity in non-stationary, unlabeled multimodal streams, enabling robust continual adaptation.
Key Results
- On six diverse multimodal tasks (TextVQA, SciVQA, StockQA, GQA, DriveLM, PMC-VQA), VDA achieved an average accuracy of 62.5%, outperforming the best unsupervised baseline (58.8%) and continual learning baseline (60.1%) by 3.7 and 2.4 percentage points respectively. The method demonstrated superior knowledge retention and transfer in dynamic environments, with significant improvements in complex scenarios such as medical imaging and scientific diagrams.
- Ablation studies confirmed VC-OT's role in maintaining visual structure, reducing cross-modal forgetting, while VMA enhanced task plasticity by focusing on visually relevant tokens. The combined approach effectively mitigates structural drift and bias, leading to consistent performance across tasks. Results indicate that VDA effectively balances stability and plasticity, crucial for real-world applications.
- Across different domains, VDA consistently outperformed baselines, especially in visually rich, complex scenes. Its ability to adapt to non-stationary data streams with minimal forgetting marks a significant step forward in unsupervised continual multimodal learning, opening avenues for autonomous, lifelong AI systems.
Significance
This work addresses a fundamental challenge in multimodal AI—how to enable models to continually learn from streaming, unlabeled data without catastrophic forgetting. By focusing on the structural integrity of visual dependence at the token level, the proposed VDA framework offers a novel solution that enhances model stability and plasticity simultaneously. It bridges the gap between static unsupervised training and dynamic real-world deployment, paving the way for more autonomous, adaptable AI systems capable of lifelong learning. The approach’s robustness and scalability make it highly relevant for applications in healthcare, robotics, and intelligent assistants, where continuous adaptation is essential.
Technical Contribution
The paper introduces VC-OT, an optimal transport-based mechanism that preserves the internal structure of visual dependencies by modeling region-aware costs and dependence stratification. This prevents visual focus drift and cross-modal attributional forgetting. Additionally, VMA dynamically modulates learning weights based on token-level visual dependence, promoting visually grounded knowledge acquisition while suppressing language bias. These innovations enable a unified framework that maintains old-task stability and facilitates new-task plasticity, a significant departure from prior methods relying solely on regularization or replay. Theoretical guarantees on structure preservation and empirical validation across multiple tasks demonstrate the framework’s effectiveness.
Novelty
This research is the first to leverage token-level visual dependence as a core signal for continual learning in multimodal models. It innovatively formulates the structural drift as an optimal transport problem, integrating region-aware costs and dependence stratification to prevent visual focus drift. The introduction of VMA for dynamic weight adjustment based on visual reliance is also novel, providing a fine-grained control mechanism for balancing stability and plasticity. These contributions significantly advance the state-of-the-art in unsupervised, non-stationary multimodal learning, addressing limitations of existing regularization and replay-based methods.
Limitations
- The VC-OT optimization involves high computational complexity, which may limit real-time deployment in large-scale systems. Further efficiency improvements are needed.
- The approach relies on accurate estimation of token-level visual dependence, which can be sensitive to noise or ambiguous visual inputs, especially in cluttered or low-quality images.
- The current framework primarily targets scenarios with moderate visual complexity; highly dynamic or multi-object scenes may require additional mechanisms for dependency modeling.
Future Work
Future research will focus on reducing computational overhead of VC-OT, possibly through approximation algorithms. Extending the framework to multi-object and multi-view scenarios will enhance robustness. Incorporating reinforcement learning strategies could further optimize the transport and adaptation processes, enabling more autonomous and scalable lifelong learning systems.
AI Executive Summary
In recent years, multimodal large language models (MLLMs) have achieved remarkable progress by integrating visual and linguistic understanding, enabling applications such as visual question answering and scientific diagram analysis. However, a critical challenge remains: how to enable these models to continually learn from streaming, unlabeled data in dynamic environments without suffering from catastrophic forgetting. Traditional supervised fine-tuning and reinforcement learning approaches require extensive labeled data and are not scalable for real-world, evolving scenarios.
This paper introduces a novel framework called Visual Dependence-Aware (VDA), designed to address the core issues of stability and plasticity in unsupervised continual learning. VDA leverages the concept of token-level visual dependence, which measures how much each token’s prediction relies on visual input. The framework comprises two main components: VC-OT and VMA. VC-OT formulates the preservation of old-task visual structures as an optimal transport problem, employing region-aware costs and dependence stratified penalties to prevent visual focus drift and cross-modal attributional forgetting. VMA dynamically adjusts the learning emphasis on tokens based on their visual dependence, promoting the acquisition of visually grounded knowledge while reducing language bias.
Extensive experiments across six diverse multimodal tasks demonstrate the effectiveness of VDA. The method achieves an average accuracy of 62.5%, surpassing existing unsupervised and continual learning baselines. It maintains high stability on previous tasks while exhibiting strong plasticity for new tasks, even in non-stationary data streams. These results highlight VDA’s potential to enable AI systems that can autonomously adapt and evolve in complex, real-world scenarios.
Overall, this work advances the frontier of multimodal continual learning, providing a theoretically grounded, practically effective solution for lifelong AI. Its ability to preserve visual structure and promote flexible learning paves the way for more intelligent, autonomous systems capable of sustained adaptation without reliance on labeled data.
Deep Analysis
Background
多模态大模型（MLLMs）结合视觉与语言能力，推动人工智能的快速发展。从早期的ViLBERT、LXMERT到后续的UNITER、VisualBERT，模型在多模态特征融合方面取得显著成就。然而，随着模型规模扩大和应用场景复杂化，持续学习成为新的瓶颈。传统方法多依赖有标签数据进行微调，面临数据获取成本高、环境动态变化带来的遗忘问题。近年来，无监督学习和连续学习技术逐渐兴起，试图在无需标注的情况下实现模型的持续演化，但大多忽视目标词的视觉依赖结构，导致跨模态遗忘和偏差问题突出。实际应用中，模型在不断变化的环境中需要保持稳定性和塑性，现有方法难以兼顾，亟需创新机制。
Core Problem
核心问题在于多模态模型在连续学习过程中，目标词的视觉依赖结构会发生漂移，导致跨模态归因能力下降，即所谓的跨模态遗忘。同时，模型在学习新任务时容易偏向语言偏置，削弱视觉基础的能力。这种结构漂移不仅影响模型性能，还限制其在实际场景中的应用。传统方法多依赖正则化或重放机制，但在非平稳、多任务环境下效果有限，难以同时保证旧任务的稳定性和新任务的塑性，成为制约多模态持续学习的关键难题。
Innovation
本研究提出以目标词的视觉依赖结构作为连续学习的核心信号，结合区域感知的最优传输模型VC-OT，保持旧任务中的视觉结构，防止遗忘。引入依赖层级惩罚，限制视觉焦点的全球漂移，确保结构的稳定性。VMA机制根据目标词的视觉依赖动态调节学习权重，强化视觉基础，抑制语言偏置。这两个机制协同工作，突破传统依赖标签或全局参数正则化的限制，实现模型在非平稳环境中的稳定塑性兼容。该框架首次系统性地将视觉依赖结构作为连续学习的核心信号，提供了新的理论基础和实践路径。
Methodology
- �� 目标定义：在无标签环境下，模型需在连续任务中保持旧能力同时学习新能力。
- �� 视觉依赖结构量化：通过对比真实与反事实视觉输入，计算每个目标词的视觉依赖值。
- �� VC-OT设计：
- 采用区域感知成本，衡量不同区域的视觉相似性，限制视觉焦点的全球漂移。
- 设计依赖层级惩罚，阻止视觉依赖转移到无关词汇。
- 通过Sinkhorn算法优化传输计划，保持视觉依赖的结构与强度。
- �� VMA机制：
- 根据目标词的视觉依赖，调节其学习权重。
- 强化视觉基础信息，抑制语言偏置。
- 结合目标损失，动态调整模型的塑性与稳定性。
- �� 训练流程：在不断变化的多模态数据流中，交替应用VC-OT保持结构，VMA增强塑性，整体优化目标为总损失。
Experiments
采用六个多模态任务（TextVQA、SciVQA、StockQA、GQA、DriveLM、PMC-VQA），每个任务包含不同类型的图像问答数据。模型在每个任务后评估在所有任务上的准确率，比较不同方法的性能。超参数包括VC-OT的区域相似性温度τD=0.05，缓冲区大小1000。对比基线包括无监督后训练方法（如ScPO、TLM）和连续学习方法（如SEEKR-MLLM、DGG）。通过消融实验验证VC-OT和VMA的贡献，分析模型在动态环境中的遗忘与迁移能力。
Results
VDA在六个任务中平均准确率达62.5%，优于最强无监督方法（58.8%）和连续学习基线（60.1%），提升幅度分别为3.7和2.4个百分点。尤其在医学和科学图表等复杂场景中表现出色，显示出优越的泛化能力。消融实验表明VC-OT显著减少跨模态遗忘，VMA提升新任务的学习塑性。整体验证了VDA在非平稳、多任务环境中的稳定性和迁移能力，为多模态持续学习提供了新思路。
Applications
该方法适用于需要模型持续适应新任务的场景，如智能助手、医疗影像分析、工业检测等。无需标注数据，降低数据准备成本，提升模型自主学习能力。未来可结合强化学习策略，优化结构保持和迁移效率，推动模型在无人监督环境中的自主演化。
Limitations & Outlook
当前VC-OT的优化计算复杂，难以在大规模实时场景中应用。模型在极端非平稳或噪声较大的环境下仍可能出现视觉结构偏差。未来需提升算法效率，增强对复杂多目标、多视角依赖关系的适应能力。
Plain Language Accessible to non-experts
想象你在经营一家工厂，每天都在生产不同的产品。工人（模型）需要记住每个产品的制作步骤（视觉依赖），但随着新产品不断加入，旧的流程可能会被遗忘或搞混。为了让工厂既能快速学会新产品，又不忘记旧的工艺，你需要一个聪明的管理系统（VDA）。这个系统会特别关注每个产品的关键步骤（目标词的视觉依赖），确保工人不会偏离正确流程（保持视觉结构），同时根据新产品的特点调整工人的注意力（VMA），让他们既能掌握新技能，又不丢失旧知识。这样，工厂就能持续高效地生产各种产品，适应市场变化。
ELI14 Explained like you're 14
想象你在学校学新东西，比如新的游戏规则，但你也要记住以前的规则。有时候，学新规则会把你以前的记忆搞混，变得不清楚。这就像你在玩游戏时，突然发现自己忘记了之前的技巧，或者新学的技巧让你偏离了原来的玩法。为了避免这种情况，你的老师（算法）设计了一套聪明的方法，帮助你既能学会新规则，又不忘记旧的。这个方法会特别关注你在游戏中用到的关键技巧（视觉依赖），确保你在学习新内容时不会偏离重点。老师还会根据你的掌握程度，调整教学内容，让你既能快速适应新游戏，又能保持原有的技能。这就像一个聪明的教练，既让你变得更厉害，又保证你不会迷失方向。
Abstract
In this paper, we explore a novel task of Multimodal Unsupervised Continual Post-Training (MU-CPT), enabling deployed MLLMs to continually evolve from streaming unlabeled data. Existing unsupervised post-training methods for MLLMs typically optimize target tokens uniformly, overlooking their heterogeneous visual dependence (VD). However, we reveal that token-level VD is crucial for MU-CPT. Specifically, its structural distortion serves as an indicator of cross-modal catastrophic forgetting, and its inherent heterogeneity acts as a compass to guide new-task learning. Leveraging this property, we propose a Visual Dependence-Aware (VDA) framework with two main components. First, Visually Constrained Optimal Transport (VC-OT) formulates the VD structural distortion of old-task VD during new-task learning as an optimal transport problem to mitigate cross-modal forgetting. By designing a region-aware ground cost and a dependence-stratified transport penalty, it prevents global shifts in visual focus while strictly prohibiting visual reliance from degenerating into language bias. Second, Visually Modulated Adaptation (VMA) exploits VD heterogeneity to emphasize visually grounded new-task learning, promoting new-task plasticity. Together, our method simultaneously maintains old-task stability and new-task plasticity during challenging MU-CPT. Extensive experiments under our MU-CPT setting validate the effectiveness of VDA.