Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models
Vary method scales vision vocabulary, achieving 78.2% ANLS on DocVQA and 36.2% on MMVet.
Key Findings
Methodology
Vary expands LVLMs' vision vocabulary by generating and integrating new vocabularies. It uses a vocabulary network and a small decoder to generate new vocabularies, which are then merged with the original CLIP vocabulary to enhance fine-grained perception.
Key Results
- Vary achieved 78.2% ANLS on DocVQA, significantly outperforming existing methods.
- In MMVet, Vary scored 36.2%, demonstrating strong performance across scenarios.
- Experiments show Vary excels in document parsing tasks, particularly in OCR and markdown conversion.
Significance
Vary addresses the vocabulary insufficiency of CLIP in non-English scenarios, enhancing LVLMs' performance in fine-grained vision tasks. This method provides new insights for the development of vision-language models.
Technical Contribution
Vary introduces a novel vision vocabulary expansion method, combining autoregressive generation and vocabulary fusion techniques, significantly enhancing LVLMs' fine-grained perception capabilities.
Novelty
Vary is the first to propose expanding LVLMs' vision vocabulary by generating new vocabularies and merging them with CLIP, addressing vocabulary insufficiency in non-English scenarios.
Limitations
- Vary still has room for improvement in handling high-resolution images, especially in complex scenarios.
- The vocabulary generation process may require substantial computational resources.
Future Work
Future research could explore more efficient vocabulary generation methods, optimize computational resource usage, and expand to more languages and vision tasks.
AI Executive Summary
Modern large vision-language models (LVLMs) often rely on CLIP as their vision vocabulary, but it falls short in tasks requiring dense and fine-grained vision perception, especially in non-English scenarios. To address this, the paper introduces Vary, an effective method to expand the vision vocabulary of LVLMs. Vary enhances the model's fine-grained perception by generating new vision vocabularies and merging them with CLIP.
The core technology of Vary includes a vocabulary network and a small decoder, which generate new vocabularies through autoregression and integrate them with the original vocabulary. Experiments demonstrate that Vary achieves 78.2% ANLS on DocVQA and 36.2% on MMVet, significantly outperforming existing methods.
While Vary performs well across scenarios, there is room for improvement in handling high-resolution images. Future research could explore more efficient vocabulary generation methods to further optimize model performance.
Deep Analysis
Background
Vision-language models have rapidly developed in recent years, with CLIP widely used as a general vision vocabulary. However, CLIP's inefficiency in non-English scenarios and high-resolution images has become a bottleneck for LVLMs.
Core Problem
CLIP's vocabulary insufficiency in non-English scenarios leads to inefficient encoding of visual information, affecting LVLMs' performance in fine-grained vision tasks.
Innovation
Vary addresses the vocabulary insufficiency of existing models in non-English scenarios by generating new vision vocabularies and merging them with CLIP, significantly enhancing fine-grained perception.
Methodology
- �� Use a vocabulary network and a small decoder to generate new vision vocabularies.
- �� Merge the new vocabularies with CLIP to expand the vision vocabulary.
- �� Freeze vocabulary networks to prevent overwriting visual knowledge.
Experiments
Experiments were conducted on DocVQA and MMVet datasets, using LLaVA-80k and LLaVA-CC665k as SFT data to evaluate Vary's performance across tasks.
Results
Vary achieved 78.2% ANLS on DocVQA and 36.2% on MMVet, significantly outperforming existing methods.
Applications
Vary can be applied to document parsing, OCR, and chart understanding tasks, particularly in non-English scenarios.
Limitations & Outlook
Vary has room for improvement in handling high-resolution images; future research could explore more efficient vocabulary generation methods.
Plain Language Accessible to non-experts
Imagine a library where CLIP is an encyclopedia covering most common knowledge, but for specific topics like non-English books, it lacks information. Vary acts like adding new chapters to this encyclopedia, specifically addressing these topics. This way, the library can better help you find the information you need.
ELI14 Explained like you're 14
Imagine you're playing a game with a robot helper that identifies items in the game. But when you enter a new level with lots of foreign text, the helper struggles. Vary is like an upgrade for this helper, allowing it to recognize more types of text and images, so you can breeze through the level!
Glossary
CLIP
A general vision vocabulary widely used in vision-language models.
CLIP is used as the original vision vocabulary in Vary.
LVLMs
Large vision-language models that combine visual and language processing capabilities.
Vary aims to enhance LVLMs' fine-grained perception capabilities.
DocVQA
A document visual question answering dataset used to evaluate models' document parsing abilities.
Vary achieved 78.2% ANLS on DocVQA.
MMVet
A multimodal evaluation tool used to test models' comprehensive performance.
Vary scored 36.2% on MMVet.
Autoregression
A method for generating sequential data by predicting the next element.
Vary uses autoregression to generate new vision vocabularies.
Open Questions Unanswered questions from this research
- 1 How to further expand vision vocabulary without increasing computational complexity?
- 2 How to optimize the vocabulary generation process to accommodate more languages and tasks?
Applications
Immediate Applications
Document Parsing
Vary can improve the accuracy of OCR and document parsing, especially in non-English scenarios.
Long-term Vision
Multilingual Support
By expanding vision vocabulary, Vary has the potential to support more languages in visual tasks, driving global applications.
Abstract
Modern Large Vision-Language Models (LVLMs) enjoy the same vision vocabulary -- CLIP, which can cover most common vision tasks. However, for some special vision task that needs dense and fine-grained vision perception, e.g., document-level OCR or chart understanding, especially in non-English scenarios, the CLIP-style vocabulary may encounter low efficiency in tokenizing the vision knowledge and even suffer out-of-vocabulary problem. Accordingly, we propose Vary, an efficient and effective method to scale up the vision vocabulary of LVLMs. The procedures of Vary are naturally divided into two folds: the generation and integration of a new vision vocabulary. In the first phase, we devise a vocabulary network along with a tiny decoder-only transformer to produce the desired vocabulary via autoregression. In the next, we scale up the vanilla vision vocabulary by merging the new one with the original one (CLIP), enabling the LVLMs can quickly garner new features. Compared to the popular BLIP-2, MiniGPT4, and LLaVA, Vary can maintain its vanilla capabilities while enjoying more excellent fine-grained perception and understanding ability. Specifically, Vary is competent in new document parsing features (OCR or markdown conversion) while achieving 78.2% ANLS in DocVQA and 36.2% in MMVet. Our code will be publicly available on the homepage.