XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models
XModBench evaluates Gemini 2.5 Pro's cross-modal consistency, revealing less than 60% accuracy in spatial and temporal reasoning.
Key Findings
Methodology
XModBench evaluates cross-modal consistency through multiple-choice questions across audio, vision, and text. It spans five task domains: perception, spatial reasoning, temporal reasoning, linguistic understanding, and external knowledge. Each question is presented in six modality configurations to ensure semantic consistency.
Key Results
- Gemini 2.5 Pro struggles with spatial and temporal reasoning tasks, achieving less than 60% accuracy, indicating significant shortcomings in these areas.
- Performance significantly drops when audio is used as the information carrier, highlighting audio modality as a weak link.
- Lower consistency is observed when vision serves as context compared to text, revealing directional imbalance.
Significance
XModBench provides a fundamental tool for evaluating and improving cross-modal capabilities, highlighting current large language models' shortcomings in truly modality-invariant reasoning. This is significant for advancing research and applications in multimodal models.
Technical Contribution
XModBench systematically covers all six modality combinations for the first time and introduces diagnostic metrics for modality disparity and directional imbalance, providing new evaluation standards for cross-modal consistency.
Novelty
XModBench is the first benchmark specifically designed to evaluate tri-modal consistency, filling a gap in existing benchmarks regarding modality consistency evaluation.
Limitations
- Models perform poorly in processing audio information, possibly due to the complexity and diversity of audio data.
- Lower consistency when vision is used as context, possibly due to the ambiguity of visual information.
Future Work
Future research can explore techniques to improve audio modality processing, enhance symmetry across different modalities, and develop more robust training strategies.
AI Executive Summary
With the development of large language models, cross-modal capabilities have become a significant research focus. However, existing benchmarks primarily assess general cross-modal question-answering abilities, neglecting issues of modality-invariant reasoning and modality-specific biases.
XModBench introduces a large-scale tri-modal benchmark to systematically measure cross-modal consistency. It includes 60,828 multiple-choice questions covering five task domains and tests across six modality combinations. Experimental results show that even the strongest model, Gemini 2.5 Pro, performs poorly in spatial and temporal reasoning, with accuracy below 60%.
These findings indicate that current large language models have a long way to go in achieving truly modality-invariant reasoning. XModBench provides a foundational tool for evaluating and improving cross-modal capabilities, with data and evaluation tools made publicly available to foster further research in this field.
Deep Analysis
Background
In recent years, cross-modal large language models (OLLMs) have made significant strides in unifying audio, vision, and text understanding. Despite expanded modality coverage, a key question remains: do these models achieve truly modality-invariant reasoning? Existing benchmarks primarily focus on cross-modal question-answering abilities, overlooking modality consistency.
Core Problem
The core problem is assessing whether OLLMs maintain consistent predictions across different modalities, i.e., modality-invariant reasoning. It remains unclear whether current models rely on modality-specific surface features rather than shared semantic representations.
Innovation
XModBench introduces a tri-modal benchmark to evaluate cross-modal consistency systematically. Its innovations include covering all six modality combinations and introducing diagnostic metrics for modality disparity and directional imbalance.
Methodology
- �� XModBench is designed as multiple-choice questions with context and candidates.
- �� Systematically permutes audio, vision, and text to generate six modality configurations.
- �� Covers five task domains, ensuring broad domain coverage and fine-grained diagnostics.
Experiments
Experiments use XModBench to evaluate various OLLMs, focusing on task competence, modality disparity, and directional imbalance. Results show strong performance in perception and linguistic tasks but poor performance in spatial and temporal reasoning.
Results
Experimental results indicate that current OLLMs achieve less than 60% accuracy in spatial and temporal reasoning tasks, with significant performance drops when audio is involved.
Applications
XModBench can be used to evaluate and improve OLLMs' cross-modal capabilities, particularly in applications requiring modality-invariant reasoning, such as multimodal human-computer interaction and autonomous driving.
Limitations & Outlook
Current models perform poorly in processing audio information, possibly due to the complexity of audio data. Lower consistency when vision is used as context, possibly due to the ambiguity of visual information.
Plain Language Accessible to non-experts
Imagine you're at a theme park with three different games: visual, auditory, and textual. Each game requires you to recognize the same thing using different senses. XModBench is like the park manager, ensuring each game follows the same rules and testing whether your senses can maintain consistent performance across different games.
ELI14 Explained like you're 14
Imagine you're playing a super cool game with sounds, pictures, and words. Your task is to find the connection between these different forms of information. XModBench is like the game's referee, helping you judge whether you've found the same answer in these different forms. Isn't that fun?
Glossary
Cross-modal
Refers to interactions involving multiple senses or forms of information, such as vision, audio, and text.
Used in the paper to describe the model's ability to handle different modality information.
Modality Consistency
The ability of a model to maintain stable predictions across different modalities.
Used to evaluate the model's performance consistency across modalities.
Gemini 2.5 Pro
An advanced cross-modal large language model focusing on unified understanding of audio, vision, and text.
Tested as one of the strongest models in the experiments.
Modality Disparity
Refers to performance differences across modalities, revealing reliance on specific modalities.
Used to diagnose performance differences across modalities.
Directional Imbalance
Refers to asymmetries in performance when context and candidate modalities are swapped.
Used to analyze performance asymmetries across modality combinations.
Open Questions Unanswered questions from this research
- 1 How to improve model performance in the audio modality? Current methods struggle with the complexity of audio data.
- 2 How to achieve truly modality-invariant reasoning? Current models still need improvement in consistency across modalities.
Applications
Immediate Applications
Multimodal Human-Computer Interaction
Improving cross-modal consistency can enhance the naturalness and accuracy of human-computer interaction systems.
Long-term Vision
Autonomous Driving
Enhancing decision-making consistency across different perception modalities can improve safety in autonomous driving systems.
Abstract
Omni-modal large language models (OLLMs) aim to unify audio, vision, and text understanding within a single framework. While existing benchmarks primarily evaluate general cross-modal question-answering ability, it remains unclear whether OLLMs achieve modality-invariant reasoning or exhibit modality-specific biases. We introduce XModBench, a large-scale tri-modal benchmark explicitly designed to measure cross-modal consistency. XModBench comprises 60,828 multiple-choice questions spanning five task families and systematically covers all six modality compositions in question-answer pairs, enabling fine-grained diagnosis of an OLLM's modality-invariant reasoning, modality disparity, and directional imbalance. Experiments show that even the strongest model, Gemini 2.5 Pro, (i) struggles with spatial and temporal reasoning, achieving less than 60% accuracy, (ii) reveals persistent modality disparities, with performance dropping substantially when the same semantic content is conveyed through audio rather than text, and (iii) shows systematic directional imbalance, exhibiting lower consistency when vision serves as context compared to text. These findings indicate that current OLLMs remain far from truly modality-invariant reasoning and position XModBench as a fundamental diagnostic tool for evaluating and improving cross-modal competence. All data and evaluation tools will be available at https://xingruiwang.github.io/projects/XModBench/.