MANTIS: Interleaved Multi-Image Instruction Tuning
MANTIS enhances multi-image vision language skills via instruction tuning, achieving 13-point improvement.
Key Findings
Methodology
MANTIS employs instruction tuning by constructing the Mantis-Instruct dataset containing 721K multi-image instruction data, training models with skills like co-reference, comparison, reasoning, and temporal understanding. The architecture integrates LLaMA-3 language model and CLIP vision encoder, using a multimodal projector to map vision embeddings to text embeddings.
Key Results
- Mantis-Idefics2 achieves state-of-the-art results on 8 multi-image benchmarks, outperforming Idefics2-8B by an average of 13 absolute points.
- On single-image benchmarks, Mantis maintains strong performance comparable to CogVLM and Emu2.
- Mantis performs consistently on held-in and held-out benchmarks, demonstrating good generalization ability.
Significance
The study shows that multi-image capabilities can be effectively enhanced through low-cost instruction tuning rather than relying on massive pre-training, offering new insights for future multimodal model improvements.
Technical Contribution
MANTIS achieves multi-image capability enhancement through instruction tuning, providing new theoretical guarantees and engineering possibilities, fundamentally differing from existing SOTA methods in dataset size and training efficiency.
Novelty
MANTIS is the first to demonstrate that multi-image capabilities can be acquired through instruction tuning, significantly reducing computational costs compared to large-scale pre-training.
Limitations
- The model may underperform in extremely complex scenarios, especially tasks requiring deep reasoning.
- Limited dataset coverage may affect performance on specific tasks.
Future Work
Future work could explore a wider range of multi-image tasks and optimize model architecture for further performance enhancement.
AI Executive Summary
In recent years, multimodal models have made significant progress in single-image vision language tasks, yet challenges remain in multi-image tasks. Existing models like OpenFlamingo and Idefics rely on massive noisy data pre-training, which is inefficient and limited in effectiveness. MANTIS employs instruction tuning, utilizing academic-level resources to construct the Mantis-Instruct dataset, successfully training models with multi-image capabilities. Experimental results show that Mantis-Idefics2 achieves state-of-the-art performance on multi-image benchmarks and maintains strong performance in single-image tasks. The study demonstrates that multi-image capabilities can be acquired through low-cost instruction tuning, offering an efficient solution. Future work will continue to optimize model architecture and expand dataset coverage to further enhance multimodal models' multi-image capabilities.
Deep Analysis
Background
Multimodal models have recently made significant progress in vision language tasks, particularly in single-image tasks. However, challenges remain in multi-image tasks, as existing models like OpenFlamingo and Idefics rely on massive noisy data pre-training, which is inefficient and limited in effectiveness.
Core Problem
Multi-image vision language tasks require models to possess co-reference, comparison, reasoning, and temporal understanding skills, which are crucial in real-world applications, yet existing models still need improvement in these tasks.
Innovation
MANTIS employs instruction tuning, constructing the Mantis-Instruct dataset to significantly enhance models' multi-image capabilities. Compared to traditional methods, instruction tuning does not require massive data pre-training, reducing computational costs.
Methodology
- �� Construct Mantis-Instruct dataset with 721K multi-image instruction data
- �� Use LLaMA-3 language model and CLIP vision encoder for model training
- �� Apply multimodal projector to map vision embeddings to text embeddings
- �� Empower models with multi-image task capabilities through instruction tuning
Experiments
Experiments use 8 multi-image benchmarks and 6 single-image benchmarks, comparing Mantis with existing models' performance. Training is conducted on 16 A100 GPUs, evaluating model performance on held-in and held-out benchmarks.
Results
Mantis-Idefics2 achieves state-of-the-art results on multi-image benchmarks, outperforming Idefics2-8B by an average of 13 absolute points. On single-image tasks, Mantis maintains strong performance comparable to CogVLM and Emu2.
Applications
MANTIS can be applied to multi-image vision language tasks such as image comparison, reasoning, and temporal understanding, suitable for applications requiring multi-image input processing.
Limitations & Outlook
The model may underperform in extremely complex scenarios, especially tasks requiring deep reasoning. Limited dataset coverage may affect performance on specific tasks.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen, needing to handle multiple ingredients simultaneously. Each ingredient requires different processing methods, like slicing, boiling, or stirring. MANTIS acts like a smart chef, capable of handling multiple ingredients at once and choosing the best processing method based on each ingredient's characteristics. Through instruction tuning, MANTIS learns to find the best solutions in multi-image tasks, just like a chef finding the best cooking method in complex dishes.
ELI14 Explained like you're 14
Hey there! Imagine you're playing a super cool game where you need to pay attention to multiple screens at once. Each screen has different tasks, like finding hidden treasures or solving puzzles. MANTIS is like a super smart game assistant that can handle all the information on multiple screens and help you find the best solutions. Through instruction tuning, MANTIS learns to find the best solutions in multi-image tasks, just like a game assistant finding the best way to beat a level!
Glossary
Instruction Tuning
A method of fine-tuning models using specific instruction datasets to enhance performance in particular tasks.
MANTIS enhances multi-image task capabilities through instruction tuning.
Multimodal Model
A model capable of processing multiple data types, such as images and text.
MANTIS is a multimodal model processing images and text.
Co-reference
The ability to understand references in natural language expressions and associate them with images.
MANTIS demonstrates strong co-reference capabilities in multi-image tasks.
Vision Encoder
A component that converts image data into embeddings that the model can process.
MANTIS uses CLIP vision encoder for image processing.
Generalization Ability
The ability of a model to perform well on unseen data or tasks.
MANTIS performs consistently on held-in and held-out benchmarks, demonstrating good generalization ability.
Open Questions Unanswered questions from this research
- 1 How to further improve model performance in extremely complex scenarios?
- 2 How to expand the dataset to cover more multi-image tasks?
Applications
Immediate Applications
Image Comparison
MANTIS can be used for quick comparison of multiple images, suitable for applications requiring analysis of image differences.
Long-term Vision
Multi-image Reasoning
MANTIS can be used for complex multi-image reasoning tasks, promoting widespread use of vision language models in real-world applications.
Abstract
Large multimodal models (LMMs) have shown great results in single-image vision language tasks. However, their abilities to solve multi-image visual language tasks is yet to be improved. The existing LMMs like OpenFlamingo, Emu2, and Idefics gain their multi-image ability through pre-training on hundreds of millions of noisy interleaved image-text data from the web, which is neither efficient nor effective. In this paper, we aim to build strong multi-image LMMs via instruction tuning with academic-level resources. Therefore, we meticulously construct Mantis-Instruct containing 721K multi-image instruction data to train a family of Mantis models. The instruction tuning empowers Mantis with different multi-image skills like co-reference, comparison, reasoning, and temporal understanding. We evaluate Mantis on 8 multi-image benchmarks and 6 single-image benchmarks. Mantis-Idefics2 can achieve SoTA results on all the multi-image benchmarks and beat the strongest multi-image baseline, Idefics2-8B by an average of 13 absolute points. Notably, Idefics2-8B was pre-trained on 140M interleaved multi-image data, which is 200x larger than Mantis-Instruct. We observe that Mantis performs equivalently well on the held-in and held-out benchmarks, which shows its generalization ability. We further evaluate Mantis on single-image benchmarks and demonstrate that Mantis also maintains a strong single-image performance on par with CogVLM and Emu2. Our results show that multi-image abilities are not necessarily gained through massive pre-training, instead, they can be gained by low-cost instruction tuning. The training and evaluation of Mantis has paved the road for future work to improve LMMs' multi-image abilities.