MiCEval: Unveiling Multimodal Chain of Thought's Quality via Image Description and Reasoning Steps
MiCEval introduces a fine-grained, multi-metric framework for automated evaluation of multimodal chain-of-thought, outperforming cosine similarity and fine-tuning metrics.
Key Findings
Methodology
MiCEval constructs a detailed annotated dataset with 903 generated and 2889 human-labeled steps, incorporating step-wise correctness, relevance, and informativeness metrics. It employs prompt engineering to guide large multimodal models (MLLMs) for automatic scoring across five core tasks: description correctness, image relevance, logical correctness, logical relevance, and informativeness. The framework evaluates both step-level and sequence-level correctness, using geometric mean fusion of multi-metric scores. Extensive experiments on four state-of-the-art MLLMs demonstrate that MiCEval metrics correlate more strongly with human judgments than existing cosine similarity or fine-tuning approaches, especially in complex reasoning scenarios.
Key Results
- On the MiCEval-HARD and NORMAL datasets, the correlation coefficients between MLLM scores and human annotations reached 0.162 and 0.208 respectively, surpassing traditional similarity metrics. The models showed persistent weaknesses in description accuracy and logical consistency, highlighting the need for further multimodal and reasoning improvements.
- The multi-metric evaluation captures nuanced aspects of reasoning quality, with scores indicating that current models often produce partially correct or irrelevant steps, especially in complex reasoning tasks. Few-shot learning did not consistently outperform zero-shot, revealing challenges in generalization.
- Using the proposed framework, MLLMs as evaluators achieved higher alignment with human preferences, demonstrating the potential for automated, scalable assessment of multimodal reasoning chains, which is crucial for advancing trustworthy AI systems.
Significance
This work addresses a critical gap in multimodal AI by providing a systematic, automated, and fine-grained evaluation framework for reasoning chains. It enhances interpretability and trustworthiness of large models, facilitating their deployment in real-world applications such as automated question answering, decision support, and educational tools. The comprehensive dataset and metrics set new standards for model assessment, enabling more targeted improvements and fostering transparency in multimodal reasoning systems.
Technical Contribution
MiCEval introduces a multi-task, prompt-guided scoring system that evaluates each reasoning step across multiple dimensions, supported by a large, human-annotated dataset. It moves beyond simple similarity metrics, integrating detailed correctness, relevance, and informativeness scores into a unified evaluation pipeline. The framework demonstrates superior correlation with human judgments, setting a new benchmark for automated multimodal reasoning assessment. It also provides insights into model weaknesses, guiding future research directions.
Novelty
This is the first comprehensive framework to perform step-wise, multi-metric evaluation of multimodal chain-of-thought answers, combining prompt engineering, multi-task annotation, and model-based scoring. Unlike prior approaches limited to final answer accuracy or single-metric similarity, MiCEval captures detailed reasoning quality, addressing the core challenge of interpretability and reliability in multimodal models.
Limitations
- The evaluation relies on prompt-based outputs from models, which may introduce variability and bias. Further robustness testing across diverse tasks and models is needed.
- While the dataset covers a broad range of reasoning scenarios, extreme or domain-specific tasks remain underrepresented, limiting generalization.
- Current metrics focus on correctness and relevance but may not fully capture subtle logical errors or factual inconsistencies, necessitating integration with symbolic or knowledge-based reasoning methods.
Future Work
Future directions include integrating symbolic reasoning modules to improve logical consistency, expanding datasets with more diverse and challenging tasks, and developing self-supervised or reinforcement learning approaches to enhance model evaluation robustness. Additionally, establishing standardized benchmarks for multimodal reasoning evaluation will accelerate progress in trustworthy AI development.
AI Executive Summary
Multimodal large language models (MLLMs) have shown remarkable capabilities in handling complex reasoning tasks involving both visual and textual data. However, evaluating the reasoning process itself remains a significant challenge. Traditional metrics like cosine similarity or manual annotation are either too simplistic or too costly, limiting scalability and reliability. Recognizing this gap, Zhou et al. introduce MiCEval, a novel evaluation framework designed to assess the quality of multimodal chain-of-thought (MCoT) answers at a granular, step-wise level.
MiCEval constructs a high-quality dataset with detailed human annotations for 903 generated and 2889 human-labeled reasoning steps. It employs prompt engineering techniques to guide multiple large models in automatically scoring each step across five key dimensions: description correctness, image relevance, logical correctness, logical relevance, and informativeness. These scores are then fused using geometric mean to produce comprehensive metrics for both individual steps and entire reasoning chains.
Extensive experiments across four state-of-the-art MLLMs reveal that MiCEval's metrics correlate more strongly with human judgments than existing approaches. The results highlight that current models still struggle with complex reasoning, especially in maintaining logical consistency and accurate image descriptions. Interestingly, few-shot learning does not always outperform zero-shot, indicating the need for further robustness improvements.
The significance of this work lies in its ability to provide a scalable, automated, and nuanced assessment of multimodal reasoning, which is vital for deploying trustworthy AI systems in real-world applications such as automated question answering, decision support, and education. By offering a detailed dataset and a multi-metric evaluation pipeline, MiCEval sets a new standard for model interpretability and reliability.
Looking ahead, future research will focus on integrating symbolic reasoning, expanding dataset diversity, and developing self-supervised evaluation methods. Overall, MiCEval advances the field by enabling more transparent, accurate, and scalable assessment of multimodal reasoning capabilities, paving the way for more trustworthy and explainable AI systems.
Deep Dive
Abstract
Multimodal Chain of Thought (MCoT) is a popular prompting strategy for improving the performance of multimodal large language models (MLLMs) across a range of complex reasoning tasks. Despite its popularity, there is a notable absence of automated methods for evaluating the quality of reasoning steps in MCoT. To address this gap, we propose Multimodal Chain-of-Thought Evaluation (MiCEval), a framework designed to assess the correctness of reasoning chains by evaluating the quality of both the description and each reasoning step. The evaluation of the description component focuses on the accuracy of the image descriptions, while the reasoning step evaluates the quality of each step as it is conditionally generated based on the preceding steps. MiCEval is built upon a fine-grained dataset with annotations that rate each step according to correctness, relevance, and informativeness. Extensive experiments on four state-of-the-art MLLMs show that step-wise evaluations using MiCEval align more closely with human judgments compared to existing methods based on cosine similarity or fine-tuning approaches. MiCEval datasets and code can be found in https://github.com/alenai97/MiCEval.