MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
Proposed MME benchmark evaluates 30 multimodal large models across 14 subtasks, measuring perception and cognition with manual instruction design.
Key Findings
Methodology
This study develops manually crafted instruction-answer pairs to prevent data leakage, covering 14 subtasks split into perception and cognition. Using concise prompts like 'please answer yes or no,' it facilitates fair, quantitative evaluation. 30 models, including GPT-4V, LLaVA, and XComposer-VL, are assessed in zero-shot settings. The evaluation employs accuracy and accuracy+ metrics, with detailed analysis of performance across object recognition, scene understanding, reasoning, and code comprehension, leveraging algorithms such as ViT, CLIP, and GPT-4V. The framework emphasizes fairness, reproducibility, and comprehensive coverage of abilities.
Key Results
- Most models excel in perception tasks, achieving up to 98.33% accuracy in object existence detection, but show significant gaps in complex reasoning and code tasks, with scores ranging from 50 to 90. GPT-4V leads in many subtasks, indicating strong multi-modal understanding.
- In fine-grained recognition, models like LLaVA and XComposer-VL outperform others, while in OCR and reasoning, the scores vary widely, highlighting the diversity in model capabilities. The evaluation reveals prevalent issues such as object hallucination and instruction misinterpretation.
- Statistical analysis confirms that models struggle with complex reasoning, and performance gaps suggest directions for targeted improvements, especially in robustness and generalization.
Significance
This benchmark provides a systematic, multi-dimensional evaluation of multimodal large models, addressing the lack of standardized, comprehensive assessment tools. It enables objective comparison, identifies strengths and weaknesses, and guides future research toward models with balanced perception and reasoning abilities. The approach enhances transparency and reproducibility, fostering progress in both academia and industry by setting clear performance standards.
Technical Contribution
The paper introduces a unified evaluation framework based on manually designed concise instructions, covering 14 diverse subtasks. It combines multiple metrics—accuracy and accuracy+—to rigorously quantify model performance. The evaluation of 30 models demonstrates the framework's effectiveness and reveals critical limitations, such as hallucination and poor instruction understanding, providing a foundation for future model development and benchmarking standards.
Novelty
This is the first comprehensive benchmark integrating perception and cognition in a unified, fair evaluation system for multimodal large models. It avoids data leakage by manual annotation, employs simple yet effective prompts, and covers a broad spectrum of abilities, setting a new standard for model assessment in this domain.
Limitations
- The current evaluation relies on static images, lacking dynamic video or audio assessment, which limits understanding of models in real-time scenarios.
- While instructions are concise, models still exhibit difficulty in following complex or multi-step commands, indicating room for improvement in instruction understanding.
- Manual data collection introduces subjective bias; future work should incorporate automated or real-world data to enhance robustness and ecological validity.
Future Work
Future directions include extending the benchmark to video and audio modalities, developing more complex instruction sets, and automating data generation to reduce bias. Additionally, integrating more nuanced metrics and real-world tasks will better reflect models' practical capabilities, fostering the development of more reliable, generalizable multimodal AI systems.
AI Executive Summary
The rapid advancement of large language models (LLMs) has spurred a new wave of multimodal large models (MLLMs), capable of integrating visual and textual information for complex reasoning tasks. Models like GPT-4V, Flamingo, and MiniGPT-4 have demonstrated emergent abilities such as poem writing based on images, few-shot learning, and advanced reasoning. However, existing evaluations often rely on limited datasets or task-specific benchmarks, which do not fully capture the broad spectrum of MLLM capabilities. This gap hampers objective comparison and hinders targeted improvements.
To address this, we introduce the Multimodal Model Evaluation (MME) benchmark, a comprehensive, fair, and scalable assessment framework. It covers 14 subtasks spanning perception—object detection, scene recognition, OCR—and cognition—commonsense reasoning, numerical calculation, text translation, code understanding. All instruction-answer pairs are manually designed to prevent data leakage and ensure consistency. The prompts are intentionally concise, enabling models to generalize and reducing the influence of prompt engineering. The evaluation employs accuracy and accuracy+ metrics, with the latter requiring both questions in a pair to be answered correctly, thus providing a stricter measure of understanding.
Applying MME to 30 state-of-the-art models, including GPT-4V, LLaVA, and XComposer-VL, reveals significant performance disparities. While most models excel in perception tasks, their abilities in reasoning and complex understanding lag behind, with scores ranging from 50 to 90. Common issues such as object hallucination, instruction misinterpretation, and reasoning failures are identified, highlighting critical areas for future research. The results underscore the importance of balanced development in perception and cognition for practical deployment.
This benchmark not only sets a new standard for systematic evaluation but also offers valuable insights into current limitations and future directions. It encourages the development of models with more robust, explainable, and generalizable multimodal capabilities, ultimately accelerating progress toward truly intelligent AI systems capable of understanding and reasoning in the real world.
Deep Analysis
Background
The evolution of multimodal AI has transitioned from early fusion techniques like simple concatenation to sophisticated models such as CLIP, Florence, and GIT2, which align visual and textual representations through contrastive learning. These models have significantly advanced tasks like image retrieval, captioning, and visual question answering. Despite these progress, the lack of a unified, comprehensive evaluation framework limits the ability to compare models objectively, especially regarding emergent abilities like reasoning, hallucination mitigation, and instruction following. Existing datasets such as COCO, VQA, and ScienceQA provide valuable benchmarks but are insufficient for holistic assessment, often suffering from data leakage and bias issues. Recent efforts focus on few-shot and zero-shot capabilities, but these are evaluated on narrow tasks, failing to reflect the full potential or limitations of current models.
Core Problem
The core challenge lies in establishing a standardized, multi-dimensional evaluation system that can accurately measure both perception and cognition in multimodal models. Traditional datasets are limited in scope and prone to data leakage, which compromises fairness. Moreover, current metrics often focus on single aspects like accuracy, neglecting the interplay between perception and reasoning. As models grow more capable, the need for a comprehensive benchmark that can evaluate diverse abilities in a fair, scalable manner becomes critical. Without such a system, progress may be hindered by inconsistent comparisons and overlooked weaknesses, impeding targeted improvements and industry adoption.
Innovation
This work introduces a novel evaluation framework—MME—that addresses these issues through several key innovations. First, it employs manually crafted instruction-answer pairs, ensuring data integrity and fairness. Second, it covers 14 diverse subtasks, from object detection to code reasoning, providing a holistic view of model capabilities. Third, the prompts are concise and standardized, promoting model generalization and reducing prompt engineering bias. Fourth, the evaluation metrics combine accuracy and a stricter accuracy+ measure, offering a nuanced understanding of performance. These innovations collectively enable a fair, comprehensive, and scalable assessment of multimodal models, setting a new benchmark in the field.
Methodology
- �� Manually design instruction-answer pairs for each subtask, avoiding data leakage
- �� Cover 14 subtasks split into perception (object recognition, OCR) and cognition (reasoning, translation)
- �� Use concise prompts like 'please answer yes or no' to standardize inputs
- �� Evaluate 30 models including GPT-4V, LLaVA, XComposer-VL, with zero-shot testing
- �� Metrics include accuracy and accuracy+; accuracy+ requires both questions answered correctly
- �� Collect data from public datasets and real photographs, ensuring diversity
- �� Analyze performance gaps, identify issues like hallucination, instruction misunderstanding
Experiments
- �� Selected 30 models with architectures ranging from vision encoders to multimodal fusion models
- �� Used manually curated datasets for each subtask, ensuring no overlap with training data
- �� Conducted zero-shot evaluations, recording accuracy and accuracy+ scores
- �� Performed ablation studies on instruction complexity and data quality
- �� Analyzed model performance across subtasks, highlighting strengths and weaknesses
- �� Results validated the framework’s fairness and robustness, revealing critical bottlenecks
Results
- �� Most models achieve high accuracy (>98%) in basic perception tasks like object existence, but performance drops significantly in reasoning tasks, with scores around 50-70
- �� GPT-4V outperforms peers in most subtasks, with scores exceeding 90 in many areas, yet still exhibits hallucination issues
- �� Identified common failure modes such as object hallucination, instruction misinterpretation, and reasoning errors
- �� Performance gaps suggest targeted directions for model improvements, especially in reasoning robustness and hallucination mitigation
Applications
- �� Enhances AI systems for autonomous vehicles, medical imaging, and intelligent assistants by providing reliable perception and reasoning benchmarks
- �� Facilitates industry adoption by offering standardized performance metrics for model selection and deployment
- �� Supports academic research in understanding model limitations and guiding architecture improvements
- �� Promotes development of more balanced models capable of real-world multi-modal understanding
Limitations & Outlook
- �� Current evaluation focuses mainly on static images, lacking video and audio modalities
- �� The manual instruction design, while fair, may not cover all real-world complexities
- �� Data collection is subjective and limited in scope; future work should incorporate automated, real-world data for broader validation
Plain Language Accessible to non-experts
Imagine a classroom where different students are tested on their ability to see, understand, and think. Some students are asked to identify objects in pictures, like 'Are there two apples?' Others are asked to solve puzzles or translate sentences. To fairly compare them, the teacher gives all students the same simple instructions: 'Answer yes or no.' Every student’s answer is recorded, and the teacher checks how many they got right. This way, we can see who is good at recognizing things, who can reason, and who needs more practice. This testing method helps us understand how smart each student (or AI model) really is, and what they need to learn next.
ELI14 Explained like you're 14
Imagine you have a bunch of different robots, and you want to see how smart they are. Some robots can look at pictures and tell you what's there, like 'Is there a cat?' Others can do math, translate languages, or even write simple code. To find out how good they are, you give each robot the same simple test: ask yes/no questions about pictures or problems. For example, 'Are there two bananas in this picture?' The robot answers 'yes' or 'no.' Then, you check how many answers are correct. Some robots do really well on simple tasks but struggle with tricky questions or understanding instructions. This test helps us see which robots are ready for real-world jobs and what they still need to learn. It’s like giving a report card to AI, so we know how to make them smarter!
Abstract
Multimodal Large Language Model (MLLM) relies on the powerful LLM to perform multimodal tasks, showing amazing emergent abilities in recent studies, such as writing poems based on an image. However, it is difficult for these case studies to fully reflect the performance of MLLM, lacking a comprehensive evaluation. In this paper, we fill in this blank, presenting the first comprehensive MLLM Evaluation benchmark MME. It measures both perception and cognition abilities on a total of 14 subtasks. In order to avoid data leakage that may arise from direct use of public datasets for evaluation, the annotations of instruction-answer pairs are all manually designed. The concise instruction design allows us to fairly compare MLLMs, instead of struggling in prompt engineering. Besides, with such an instruction, we can also easily carry out quantitative statistics. A total of 30 advanced MLLMs are comprehensively evaluated on our MME, which not only suggests that existing MLLMs still have a large room for improvement, but also reveals the potential directions for the subsequent model optimization. The data are released at the project page https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Evaluation.