Beyond Final Answers: CRYSTAL Benchmark for Transparent Multimodal Reasoning Evaluation

In recent years, multimodal large language models have made significant progress in vision-language tasks. These models, which integrate pretrained visual encoders with large language models, have excelled in complex tasks. For example, the MathVista dataset consolidates diverse mathematical reasoning tasks, while RealWorldQA challenges models with spatial understanding in real-world images. However, existing evaluation methods primarily focus on the accuracy of final answers, neglecting the transparency and logic of the reasoning process. This limitation makes it difficult to distinguish whether a model arrives at an answer through shortcuts or genuine understanding and reasoning. Therefore, evaluating the transparency of multimodal reasoning has become an urgent issue to address.

Existing evaluation methods for multimodal large language models primarily focus on the accuracy of final answers, neglecting the transparency of the reasoning process. This limitation makes it difficult to distinguish whether a model arrives at an answer through shortcuts or genuine understanding and reasoning. Moreover, existing evaluation methods fail to identify systematic flaws in the reasoning process, such as the 'cherry-picking' phenomenon and issues with the order of reasoning chains. These problems may lead to poor performance in practical applications, failing to meet the requirements for transparency and reliability.

The core innovation of the CRYSTAL benchmark lies in its ability to evaluate the transparency of multimodal reasoning. First, CRYSTAL generates reference reasoning steps through a Delphi-inspired pipeline, validated via semantic clustering and human quality gates. This method ensures the diversity and high quality of reference reasoning steps. Second, CRYSTAL introduces two complementary metrics: Match F1 and Ordered Match F1, which assess step-level precision and recall, and the order of reasoning chains, respectively. This evaluation method allows for a fine-grained analysis of model performance in the reasoning process, revealing systematic flaws. Finally, the CRYSTAL benchmark not only serves as an evaluation tool but also provides new insights for model training and improvement through the Causal Process Reward (CPR) and CPR-Curriculum.

The evaluation method of the CRYSTAL benchmark includes the following steps:

• �� Reference Generation: Generate reference reasoning steps through a Delphi-inspired pipeline, using four independent multimodal large language models to generate trajectories, validated via semantic clustering and human quality gates.
• �� Metric Design: Introduce two complementary metrics: Match F1 and Ordered Match F1, which assess step-level precision and recall, and the order of reasoning chains, respectively.
• �� Model Evaluation: Evaluate 20 multimodal large language models, including some commercial frontier systems not used during benchmark construction, revealing systematic flaws in reasoning transparency.
• �� Reward Design: Propose the Causal Process Reward (CPR), which multiplicatively couples answer correctness with step-level alignment, and CPR-Curriculum, which progressively increases reasoning difficulty during training.

The experimental design includes evaluating 20 multimodal large language models, 16 of which are open-source models and 4 are commercial models. The datasets used include MathVision, ScienceQA-IMG, RealWorldQA, MMVP, and PLOTQA. The metrics used in the experiments include Match F1 and Ordered Match F1, which assess model performance in reasoning transparency. The experiments also include ablation studies to test the impact of different sentence encoders and thresholds on evaluation results. Through these experiments, the systematic flaws in reasoning transparency in existing models are revealed, and the effectiveness of the CRYSTAL benchmark is validated.

The experimental results of the CRYSTAL benchmark reveal systematic flaws in reasoning transparency in existing models. First, there is a universal 'cherry-picking' phenomenon where models' precision significantly exceeds recall. For instance, GPT-5 has a precision of 0.925 but a recall of only 0.479. Second, there is a significant divergence between accuracy and reasoning transparency. GPT-5 ranks highest in accuracy (57.99%) but only eighth in Match F1 (0.612). Finally, using Ordered Match F1, it is found that no model maintains more than 60% of matched steps in the correct order. These results indicate significant deficiencies in the transparency and logic of the reasoning process in existing models.

The application scenarios of the CRYSTAL benchmark include the evaluation and improvement of multimodal large language models. By assessing model performance in reasoning transparency, the CRYSTAL benchmark helps developers identify systematic flaws and provides directions for model improvement. Additionally, the CRYSTAL benchmark can be used to train new multimodal large language models, improving their reasoning capabilities through the Causal Process Reward (CPR) and CPR-Curriculum. In the industry, the CRYSTAL benchmark can be used to evaluate and improve the performance of multimodal large language models in practical applications, enhancing their transparency and reliability.

The limitations of the CRYSTAL benchmark include the complexity of the evaluation process, which may lead to time-consuming evaluations, especially when dealing with large-scale datasets. Additionally, the generation of reference reasoning steps relies on multimodal large language models, which may introduce model bias. The calculation of Ordered Match F1 may impose overly strict requirements on model ordering, leading to unfair evaluations for some models. Future research directions include optimizing evaluation efficiency, reducing time consumption, and improving methods for generating reference reasoning steps.