TouchStone: Evaluating Vision-Language Models by Language Models
TouchStone evaluates LVLMs using strong LLMs like GPT-4, covering five abilities and 27 subtasks.
Key Findings
Methodology
TouchStone employs a comprehensive visual dialogue dataset covering five abilities and 27 subtasks. It uses detailed image annotations to convert multimodal inputs into forms understandable by LLMs, enabling automated evaluation with powerful LLMs like GPT-4.
Key Results
- GPT-4 effectively evaluates dialogue quality, aligning with human preferences, demonstrating its capability in identifying hallucination issues.
- LLaVA and mPLUG-Owl excel in visual storytelling, showing advantages in the SFT stage.
- Qwen-VL excels in text recognition and chart analysis, benefiting from high-resolution inputs and multi-task learning.
Significance
This research addresses high labor costs and difficulty in quantifying hallucination issues in LVLM evaluation, providing a benchmark for building stronger LVLMs.
Technical Contribution
TouchStone offers a new automated method for evaluating LVLM dialogue capabilities, leveraging LLMs' text capabilities to assess multimodal dialogue quality without human intervention.
Novelty
This is the first use of powerful LLMs like GPT-4 to evaluate LVLM dialogue quality, overcoming limitations of traditional methods and providing a more comprehensive capability assessment.
Limitations
- The current dataset focuses more on basic abilities, lacking comprehensive coverage of advanced abilities.
- Weak multi-image analysis affects recognition accuracy.
Future Work
Future work will expand dataset coverage, enhance advanced ability evaluation, and explore more complex multimodal interaction scenarios.
AI Executive Summary
In recent years, large vision-language models (LVLMs) have made significant progress in connecting visual receptors with large language models (LLMs). However, existing evaluations mainly focus on recognition and reasoning abilities, lacking direct assessment of conversational skills and visual storytelling abilities. To address this, the paper proposes an evaluation method using strong LLMs as judges to comprehensively evaluate various abilities of LVLMs. Firstly, a comprehensive visual dialogue dataset named TouchStone is constructed, covering five major abilities and 27 subtasks. This dataset not only covers fundamental recognition and comprehension but also extends to literary creation. By integrating detailed image annotations, the multimodal input content is effectively transformed into a form understandable by LLMs, enabling the use of advanced LLMs to directly evaluate the quality of multimodal dialogue without human intervention. Through validation, it is demonstrated that powerful LVLMs like GPT-4 can effectively score dialogue quality using their textual capabilities alone, aligning with human preferences. The work aims to serve as a touchstone for LVLM evaluation and pave the way for building stronger LVLMs.
Deep Analysis
Background
In recent years, with the development of large language models (LLMs), vision-language models (LVLMs) have been increasingly applied in multimodal dialogues. However, existing evaluation methods mainly focus on recognition and reasoning abilities, lacking comprehensive assessment of conversational skills and visual storytelling abilities.
Core Problem
Current LVLM evaluation methods rely heavily on manual evaluation, which is costly and difficult to scale. Additionally, hallucination issues are often overlooked in current evaluations, necessitating an automated evaluation technique to provide objective and efficient assessments.
Innovation
TouchStone constructs a comprehensive visual dialogue dataset covering five abilities and 27 subtasks, using detailed image annotations to convert multimodal inputs into forms understandable by LLMs, enabling automated evaluation with powerful LLMs like GPT-4.
Methodology
- �� Construct the TouchStone dataset covering five abilities and 27 subtasks.
- �� Use detailed image annotations to convert multimodal inputs into text forms.
- �� Employ GPT-4 to evaluate dialogue quality, avoiding human intervention.
- �� Eliminate scoring position bias through position balancing.
Experiments
The experimental design includes using the TouchStone dataset for evaluation, comparing different LVLMs' performance across five abilities. GPT-4 is used as a judge to score different models' responses, and results are compared with human evaluations for validation.
Results
Results show that GPT-4 effectively evaluates dialogue quality, aligning with human preferences. LLaVA and mPLUG-Owl excel in visual storytelling, while Qwen-VL performs well in text recognition and chart analysis.
Applications
The TouchStone method can be used to evaluate LVLM performance in multimodal dialogues, helping developers identify model strengths and weaknesses, guiding model improvement and optimization.
Limitations & Outlook
The current dataset focuses more on basic abilities, lacking comprehensive coverage of advanced abilities. Additionally, weak multi-image analysis affects recognition accuracy. Future work will expand dataset coverage and enhance advanced ability evaluation.
Plain Language Accessible to non-experts
Imagine you're in a kitchen cooking, and TouchStone is like a smart assistant that not only recognizes ingredients but also suggests cooking methods based on your needs. It understands the dish you want to make by observing the color, shape, and arrangement of ingredients, providing detailed step-by-step guidance. This assistant can recognize different ingredients and adjust recipes based on your taste preferences. Even if you want to try new dishes, it can offer creative suggestions to help you prepare a delicious meal.
ELI14 Explained like you're 14
Imagine playing a super cool game, and TouchStone is like your game assistant. It recognizes various elements in the game, like enemies, items, and maps, and provides you with the best strategies. It not only helps you defeat enemies but also gives personalized advice based on your gaming style. Like an all-knowing game guide, it helps you win the game and become the ultimate player!
Glossary
Large Vision-Language Models
Models combining visual and language capabilities, able to understand and generate multimodal content.
Used for evaluating multimodal dialogue capabilities.
GPT-4
A powerful language model developed by OpenAI, known for its exceptional text generation and understanding capabilities.
Used for evaluating dialogue quality.
TouchStone
A comprehensive method for evaluating LVLMs using LLMs for automated assessment.
Used for evaluating multimodal dialogue capabilities.
Hallucination
Content generated by models that does not align with the input, potentially affecting evaluation results.
Identifying hallucination issues in evaluation.
Visual Storytelling
The ability to create literary content based on visual input.
Evaluating LVLMs' creative capabilities.
Open Questions Unanswered questions from this research
- 1 How to enhance advanced ability evaluation without increasing human costs?
- 2 How to better identify and quantify hallucination issues in models?
Applications
Immediate Applications
Multimodal Dialogue Evaluation
Developers can use TouchStone to evaluate LVLM dialogue capabilities, identifying model strengths and weaknesses.
Long-term Vision
Smart Assistant Development
Utilize TouchStone evaluation results to develop more intelligent multimodal interaction systems, enhancing user experience.
Abstract
Large vision-language models (LVLMs) have recently witnessed rapid advancements, exhibiting a remarkable capacity for perceiving, understanding, and processing visual information by connecting visual receptor with large language models (LLMs). However, current assessments mainly focus on recognizing and reasoning abilities, lacking direct evaluation of conversational skills and neglecting visual storytelling abilities. In this paper, we propose an evaluation method that uses strong LLMs as judges to comprehensively evaluate the various abilities of LVLMs. Firstly, we construct a comprehensive visual dialogue dataset TouchStone, consisting of open-world images and questions, covering five major categories of abilities and 27 subtasks. This dataset not only covers fundamental recognition and comprehension but also extends to literary creation. Secondly, by integrating detailed image annotations we effectively transform the multimodal input content into a form understandable by LLMs. This enables us to employ advanced LLMs for directly evaluating the quality of the multimodal dialogue without requiring human intervention. Through validation, we demonstrate that powerful LVLMs, such as GPT-4, can effectively score dialogue quality by leveraging their textual capabilities alone, aligning with human preferences. We hope our work can serve as a touchstone for LVLMs' evaluation and pave the way for building stronger LVLMs. The evaluation code is available at https://github.com/OFA-Sys/TouchStone.