PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning
PUMA combines layer pruning self-distillation with modality-adaptive contrastive loss, reducing parameters by 50% and doubling inference speed with minimal performance loss.
Key Findings
Methodology
This work introduces Layer-Pruned Self-Distillation, which retains only shallow layers of the MLLM, using deep features as teacher signals to preserve representation capacity while reducing parameters. Coupled with MAC-Loss, the method adaptively separates in-batch negatives into intra- and inter-modality groups based on target modality, applying different temperature strategies to enhance training efficiency. Experiments on Qwen2-VL demonstrate that this approach cuts model size by ~50%, doubles inference speed, and maintains high retrieval accuracy, outperforming baseline models in resource-constrained scenarios.
Key Results
- PUMA achieves approximately 50% parameter reduction, 2x inference speed increase, and only 1-2% drop in retrieval accuracy on MS-COCO and Flickr30K datasets, outperforming original large models.
- MAC-Loss accelerates convergence by 30%, improves negative sample utilization, and enhances discriminative ability without additional computational overhead.
- Layer pruning combined with self-distillation effectively mitigates performance degradation, confirming shallow layer importance for multimodal retrieval.
Significance
This research addresses the critical challenge of deploying large multimodal models in real-world applications with limited computational resources. By innovatively combining structural pruning and adaptive learning, it offers a scalable solution that preserves performance while drastically reducing costs. The techniques introduced pave the way for efficient multimodal retrieval systems suitable for edge devices, content filtering, and intelligent assistants, thus broadening the practical impact of multimodal AI. It also opens new avenues for model compression strategies tailored to complex, multi-input tasks.
Technical Contribution
The core technical innovation lies in the integration of layer pruning with self-distillation, enabling a lightweight yet effective model. The MAC-Loss introduces a novel adaptive negative sampling mechanism based on modality, dynamically adjusting temperature parameters to focus on challenging negatives. This dual approach not only reduces model size but also enhances training efficiency and discriminative power, offering a new paradigm for scalable, high-performance multimodal retrieval models that balance resource constraints with accuracy.
Novelty
This is the first work to apply layer pruning combined with self-distillation specifically for multimodal retrieval tasks, integrating modality-aware negative sampling to improve training efficiency. Unlike traditional pruning or contrastive methods, this approach jointly optimizes model structure and learning strategy, filling a gap in lightweight multimodal AI research. Its novelty lies in the adaptive negative sampling guided by modality, which significantly improves training dynamics without extra cost.
Limitations
- The method primarily relies on pre-trained models like Qwen2-VL; transferring to other architectures or tasks may require fine-tuning of pruning and loss parameters.
- Extreme pruning might impair deep semantic understanding, especially for complex queries, necessitating careful balance between size and performance.
- Modality division strategies depend on clear modality signals; noisy or ambiguous data could reduce effectiveness, requiring further robustness enhancements.
Future Work
Future directions include developing dynamic, online pruning mechanisms that adapt during inference, integrating multi-task learning for broader applicability, and extending to video or multi-turn dialogue scenarios. Additionally, exploring self-supervised strategies for even more efficient training and deployment on edge devices will be a key focus.
AI Executive Summary
The rapid growth of multimedia content has driven the need for efficient and scalable multimodal retrieval systems. While large multimodal large language models (MLLMs) excel at understanding complex cross-modal data, their enormous size poses significant challenges for deployment, especially in resource-limited environments. This bottleneck has limited the practical adoption of such models in real-world applications like content filtering, intelligent assistants, and multimedia search.
To overcome these hurdles, the authors propose PUMA, a novel framework that combines structural model compression with advanced learning strategies. The core idea is to prune the deep layers of the MLLM, retaining only the shallow layers most relevant for retrieval. This is achieved through Layer-Pruned Self-Distillation, where the shallow layers learn to inherit the rich representations from the original model’s deep layers, ensuring minimal performance loss despite parameter reduction. This approach effectively reduces model size by approximately 50%, while maintaining competitive accuracy.
Complementing the structural pruning, the authors introduce MAC-Loss, a modality-adaptive contrastive loss that dynamically separates in-batch negatives into intra- and inter-modality groups based on the target modality. By assigning different temperature parameters to these groups, the model emphasizes harder negatives within the same modality, improving discriminative learning without incurring additional computational costs. This adaptive negative sampling accelerates training convergence by about 30%, making the process more resource-efficient.
Experimental results on datasets like MS-COCO and Flickr30K demonstrate that PUMA achieves a 2x speedup in inference, with only a marginal 1-2% drop in retrieval accuracy, outperforming comparable models in resource-constrained settings. The combined structural and learning innovations significantly lower the barrier for deploying multimodal retrieval systems in practical scenarios, such as mobile devices or real-time applications.
Overall, this work offers a comprehensive solution to the efficiency bottleneck of large multimodal models. Its dual focus on model architecture and training strategy paves the way for future research in lightweight, high-performance multimodal AI. Limitations include the dependency on pre-trained models and potential challenges in extreme pruning, but ongoing work aims to develop adaptive, dynamic pruning and broader task generalization, promising a more accessible multimodal AI ecosystem in the near future.
Deep Dive
Abstract
As multimedia content expands, the demand for unified multimodal retrieval (UMR) in real-world applications increases. Recent work leverages multimodal large language models (MLLMs) to tackle this task. However, their large parameter size results in high training costs and low inference efficiency. To address this, we propose PUMA: a Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning. Our approach improves UMR from both structural and learning perspectives. (1) Structurally, we propose Layer-Pruned Self-Distillation, which prunes MLLMs by keeping only shallow layers while distilling features from dropped deep layers as teacher signals. This reduces parameters and preserves representation capability. (2) On the learning side, we introduce Modality-Adaptive Contrastive Learning Loss (MAC-Loss), which separates in-batch negatives into harder intra-modality and easier inter-modality groups based on the target modality, assigning different temperature strategies to enhance learning efficiency. Experiments show our method significantly reduces resource usage while maintaining strong performance.