GptGet

MultiModal Fine-tuning with Synthetic Captions

TL;DR

MultiModal Fine-tuning with Synthetic Captions significantly improves performance on 13 benchmarks, especially in few-shot learning.

cs.CV 🔴 Advanced 2026-01-29 12 views
Shohei Enomoto Shin'ya Yamaguchi
AI Reader Arxiv Page Download PDF
multimodal learning deep learning synthetic captions few-shot learning image classification

Key Findings

Methodology

This study proposes a method using Multimodal Large Language Models (MLLMs) to generate synthetic image captions, transforming unimodal datasets into multimodal ones. Carefully designed prompts, incorporating class labels and domain context, produce high-quality captions. A supervised contrastive loss function encourages clustering of same-class representations during fine-tuning. A novel inference technique leverages class-averaged text embeddings from multiple synthetic captions per image.

Key Results

  • The method outperforms baseline methods across 13 image classification benchmarks, with significant improvements in few-shot learning scenarios. For example, in 4-shot learning, the average accuracy reached 55.98%, outperforming FLYP by 3.41 percentage points.
  • In standard fine-tuning tasks using ResNet-50 as the backbone, the method outperforms all baseline methods across all 12 datasets, with an average accuracy gain of 4.69 percentage points.
  • In scenarios without model training, classification using synthetic captions outperforms all fine-tuned methods in low-shot settings.

Significance

This study introduces a multimodal fine-tuning method that significantly improves performance on downstream tasks, particularly in data-constrained scenarios. By generating synthetic captions, the research overcomes the limitations of traditional fine-tuning methods that rely on unimodal datasets, allowing models to better utilize the rich representations learned during multimodal pre-training. This method establishes a new paradigm for dataset enhancement, bridging the gap between multimodal pre-training and fine-tuning.

Technical Contribution

Technical contributions include a novel dataset enhancement method that transforms unimodal datasets into multimodal ones by generating synthetic captions. Additionally, a supervised contrastive loss function is developed, explicitly using class label information to guide the alignment of semantically similar images. These innovations enable models to better leverage the advantages of multimodal pre-training, particularly in few-shot learning scenarios.

Novelty

This method is the first to use Multimodal Large Language Models for generating synthetic captions to enhance the multimodal characteristics of unimodal datasets. Compared to previous work, this innovation provides richer information for fine-tuning by generating diverse captions, improving classification performance.

Limitations

  • The method relies on the performance of MLLMs for generating synthetic captions, which may produce inaccurate descriptions in specific domains or categories.
  • Generating and processing synthetic captions may increase computational resource consumption when handling very large datasets.

Future Work

Future research directions include exploring more efficient methods for generating synthetic captions to reduce computational resource consumption. Additionally, the method's application to more downstream tasks and further improving the quality and diversity of generated captions can be investigated.

AI Executive Summary

In deep learning, there is a significant gap between pre-training and fine-tuning. While pre-training has shifted from unimodal to multimodal learning, fine-tuning predominantly relies on unimodal datasets, limiting the advantages of rich pre-trained representations. To address this issue, researchers propose a novel method using Multimodal Large Language Models to generate synthetic image captions, transforming unimodal datasets into multimodal ones. Carefully designed prompts, incorporating class labels and domain context, produce high-quality captions. A supervised contrastive loss function encourages clustering of same-class representations during fine-tuning. A novel inference technique leverages class-averaged text embeddings from multiple synthetic captions per image.

Experimental results demonstrate that this method outperforms baseline methods across 13 image classification benchmarks, with significant improvements in few-shot learning scenarios. For example, in 4-shot learning, the average accuracy reached 55.98%, outperforming FLYP by 3.41 percentage points. This method establishes a new paradigm for dataset enhancement, bridging the gap between multimodal pre-training and fine-tuning.

However, the method relies on the performance of MLLMs for generating synthetic captions, which may produce inaccurate descriptions in specific domains or categories. Additionally, generating and processing synthetic captions may increase computational resource consumption when handling very large datasets. Future research directions include exploring more efficient methods for generating synthetic captions and applying the method to more downstream tasks.

Deep Analysis

Background

In recent years, the field of deep learning has shifted from unimodal to multimodal learning. Traditional unimodal learning primarily relies on cross-entropy loss training with images and categorical labels, while multimodal learning leverages image-caption pairs through contrastive learning. This shift has significantly enhanced models' visual understanding and representation capabilities. However, despite advances in pre-training, fine-tuning still predominantly relies on traditional unimodal datasets, leading to a significant gap between pre-training and fine-tuning.

Core Problem

The core problem is the gap between multimodal pre-training and unimodal fine-tuning. Despite the rich representations provided by multimodal pre-training, models cannot fully utilize these representations during fine-tuning due to the unimodal nature of datasets. This issue is particularly pronounced in few-shot learning scenarios, where models easily overfit to the task-specific distribution, preventing them from fully benefiting from the advantages of multimodal pre-training.

Innovation

The core innovation of this study is the proposal of a method using Multimodal Large Language Models to generate synthetic image captions, transforming unimodal datasets into multimodal ones. Carefully designed prompts, incorporating class labels and domain context, produce high-quality captions. Additionally, a supervised contrastive loss function is introduced to encourage clustering of same-class representations during fine-tuning. A novel inference technique leverages class-averaged text embeddings from multiple synthetic captions per image.

Methodology

  • �� Use Multimodal Large Language Models to generate synthetic captions, transforming unimodal datasets into multimodal ones.

  • �� Design prompts incorporating class labels and domain context to generate high-quality captions.

  • �� Introduce a supervised contrastive loss function to encourage clustering of same-class representations during fine-tuning.

  • �� Propose a novel inference technique leveraging class-averaged text embeddings from multiple synthetic captions per image.

Experiments

Experiments were conducted on 13 image classification benchmarks, including standard fine-tuning and few-shot learning scenarios. ResNet-50 and ViT-B/32 were used as backbone networks to evaluate the method's performance. Results show that the method outperforms all baseline methods across all benchmarks, with significant improvements in few-shot learning scenarios.

Results

In standard fine-tuning tasks using ResNet-50 as the backbone, the method outperforms all baseline methods across all 12 datasets, with an average accuracy gain of 4.69 percentage points. In few-shot learning scenarios, the average accuracy in 4-shot learning reached 55.98%, outperforming FLYP by 3.41 percentage points.

Applications

The method is applicable to scenarios requiring image classification in data-constrained situations, such as medical image analysis and remote sensing image classification. By generating synthetic captions, models can better utilize the advantages of multimodal pre-training, improving classification performance.

Limitations & Outlook

The method relies on the performance of MLLMs for generating synthetic captions, which may produce inaccurate descriptions in specific domains or categories. Additionally, generating and processing synthetic captions may increase computational resource consumption when handling very large datasets.

Plain Language Accessible to non-experts

Imagine you're in a library with many books, but each book lacks a cover or summary. You can only judge its category by its content. Now, imagine you have an assistant who quickly skims each book and writes a short summary for you, making it easier to determine the book's category. This is the core idea of the study: by generating synthetic captions, each image is given a brief description, enabling the model to better determine the image's category. Even with a small number of images, the model can classify more accurately because it can use this additional information to make more informed decisions.

ELI14 Explained like you're 14

Imagine you're playing a game where you have to guess the category of pictures. Usually, you can only see the pictures, but now you have an assistant who writes a little story for each picture to help you understand it better. It's like giving each picture a little label, making it easier for you to guess its category. This study does just that by generating little stories (synthetic captions) for pictures, making computers smarter at guessing picture categories. Even with only a few pictures, the computer does well because it has more information to help it decide.

Glossary

Multimodal Large Language Models (MLLMs)

Models capable of processing multiple data modalities, such as text and images.

Used to generate synthetic captions, transforming unimodal datasets into multimodal ones.

Supervised Contrastive Loss

A loss function that uses class label information to guide the alignment of semantically similar images.

Encourages clustering of same-class representations during fine-tuning.

Synthetic Captions

Image descriptions generated by Multimodal Large Language Models to enhance the multimodal characteristics of datasets.

Multiple synthetic captions are generated for each image to improve classification performance.

Few-shot Learning

The ability to learn and classify with only a small number of training samples.

The method shows significant improvements in few-shot learning scenarios.

Class-averaged Text Embeddings

Text embeddings averaged from multiple synthetic captions for each class.

Used in a novel inference technique to improve classification accuracy.

Open Questions Unanswered questions from this research

  • 1 How to improve the quality and diversity of synthetic captions without increasing computational resource consumption?
  • 2 How does the method perform on larger datasets? Are there more efficient methods for generating synthetic captions?

Applications

Immediate Applications

Medical Image Analysis

By generating synthetic captions, the method helps models classify more accurately on limited medical image data.

Long-term Vision

Remote Sensing Image Classification

Applying the method in remote sensing image classification to improve performance in data-constrained situations.

Abstract

In this paper, we address a fundamental gap between pre-training and fine-tuning of deep neural networks: while pre-training has shifted from unimodal to multimodal learning with enhanced visual understanding, fine-tuning predominantly remains unimodal, limiting the benefits of rich pre-trained representations. To bridge this gap, we propose a novel approach that transforms unimodal datasets into multimodal ones using Multimodal Large Language Models (MLLMs) to generate synthetic image captions for fine-tuning models with a multimodal objective. Our method employs carefully designed prompts incorporating class labels and domain context to produce high-quality captions tailored for classification tasks. Furthermore, we introduce a supervised contrastive loss function that explicitly encourages clustering of same-class representations during fine-tuning, along with a new inference technique that leverages class-averaged text embeddings from multiple synthetic captions per image. Extensive experiments across 13 image classification benchmarks demonstrate that our approach outperforms baseline methods, with particularly significant improvements in few-shot learning scenarios. Our work establishes a new paradigm for dataset enhancement that effectively bridges the gap between multimodal pre-training and fine-tuning. Our code is available at https://github.com/s-enmt/MMFT.

cs.CV