In-Context Collapse in Vision-Language Models and How to Mitigate it?

The evolution of vision-language models (VLMs) has been marked by breakthroughs in few-shot learning, exemplified by models like Frozen and Flamingo, which leverage large-scale pretraining and interleaved visual-text attention mechanisms. Early research focused on the capacity of these models to generalize from limited demonstrations without fine-tuning, emphasizing the importance of pretraining and attention architectures. As models scaled, their ability to handle many demonstrations was expected to improve monotonically. However, empirical observations revealed that performance often plateaued or declined with more examples, indicating underlying stability issues. Mechanistic interpretability studies uncovered that visual information becomes linearly accessible only in deep layers, while continual learning experiments demonstrated catastrophic forgetting during fine-tuning. Despite these insights, the precise internal causes of in-context failure remained elusive, especially regarding the role of the fusion interface. This gap motivated the current work, which aims to causally localize and repair the collapse phenomenon, bridging the interpretability, robustness, and continual learning literatures.

The core problem addressed is the in-context collapse observed in many multimodal models, where adding more demonstrations leads to a sharp decline in accuracy, sometimes below chance. This undermines the fundamental premise of in-context learning—improving with more data—posing a significant obstacle for deploying models in dynamic, real-world environments. Existing solutions, such as retrieval augmentation or multi-task fine-tuning, do not directly target the internal mechanisms causing collapse. The challenge lies in identifying the exact internal locus where the integration of visual and linguistic information fails, and developing a lightweight, transferable intervention that can prevent collapse across diverse tasks and models. Without such solutions, the reliability and scalability of multimodal AI remain limited.

The key innovation is the causal localization of in-context collapse to the vision-language fusion pathway, specifically at the connector and early/mid transformer layers. The authors introduce a parameter-matched lesion-and-rescue methodology, which confirms the causal role of this interface. Building on this, they develop CircA, a single, synthetic-task-trained adapter that, when inserted into the fusion pathway, confers cross-task resistance to collapse. Unlike traditional fine-tuning or retrieval-based methods, CircA operates as a lightweight, mechanism-based intervention, enabling models to maintain accuracy as demonstrations increase. This approach bridges interpretability and robustness, offering a new paradigm for mechanism-guided model repair that is both efficient and transferable.

• �� Define the VLM architecture, including visual encoder, connector, and transformer layers.
• �� Design contamination-free synthetic tasks and natural image classification benchmarks to isolate genuine in-context learning.
• �� Conduct parameter-matched lesion experiments by inserting small adapters at the connector and early/mid layers, measuring impact on collapse.
• �� Verify causal localization by observing whether adapters at these points restore or impair accuracy.
• �� Develop CircA: train a single adapter on synthetic tasks, then insert into the fusion path of various models.
• �� Evaluate across multiple datasets (CIFAR, Fashion) and model scales (0.5B–11B parameters), measuring accuracy and collapse resistance.
• �� Perform ablation studies on adapter placement, capacity, and transferability to unseen tasks.

Experiments involve synthetic contamination-free tasks and real datasets like CIFAR and Fashion, with varying demonstration counts (0-16 shots). The adapters are inserted at different layers to causally test their effect on collapse. Metrics include accuracy, collapse rate, and transferability. The models tested include open models (0.5B–11B) and the frontier Claude Sonnet 4.5. Ablation studies compare early/mid vs. late adapter placement, capacity effects, and synthetic vs. natural tasks. The experiments demonstrate that adapters at the fusion interface significantly mitigate collapse and transfer well across tasks, validating the causal localization and the effectiveness of CircA.

Adapters at the connector and early/mid layers restore the 16-shot accuracy from 0.39 to 0.91 on synthetic tasks, with collapse resistance rising from chance (~0.25) to over 0.70 on CIFAR and Fashion datasets. The transferability of CircA is confirmed by its success in unseen tasks, outperforming late-layer adapters which fail or worsen performance. The experiments also reveal a dissociation: late-layer adapters excel in weight-based consolidation but do not prevent collapse, indicating different internal roles. The results establish that the fusion interface is the critical locus for in-context stability, and that a single synthetic-trained adapter can confer cross-task robustness.

The proposed mechanism can be integrated into multimodal AI systems for real-time visual question answering, adaptive robotics, and interactive assistants, where robustness to diverse demonstrations is essential. Its lightweight nature allows deployment in resource-constrained environments, enabling models to adapt online without catastrophic forgetting. Long-term, this approach could facilitate continual learning in multimodal systems, supporting lifelong adaptation in dynamic environments such as autonomous vehicles, industrial robots, and personalized AI assistants, reducing the need for costly retraining.

The current validation focuses on synthetic and limited natural tasks; effectiveness in large-scale, real-world scenarios remains to be tested. The adapter's training on synthetic data may limit its generalization to highly diverse or domain-specific tasks. The mechanism primarily addresses fusion interface collapse, but other sources like pretraining biases or model capacity constraints are not fully explored. Further research is needed to integrate multiple mechanisms for comprehensive robustness, and to optimize online, dynamic adaptation strategies.