Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models
BloomBench evaluates VLM cognition across six Bloom levels using 7,747 bilingual items and reports 98.45% audited quality.
Key Findings
Methodology
BloomBench maps Remember, Understand, Apply, Analyze, Evaluate, and Create to 106 visual-question-answer task types. Gemini 2.5 Pro generates culturally varied scenarios and VQA items; an instruction-tuned model creates four-option MCQs and Modern Standard Arabic translations. LLM-as-a-judge filtering, Gemini 3 Pro auditing, and human verification provide hybrid quality assurance.
Key Results
- The benchmark contains 7,747 bilingual image-question-answer pairs: 2,948 Remember, 1,592 Understand, 499 Apply, 1,431 Analyze, 592 Evaluate, and 685 Create items across 106 taxonomy leaves.
- Zero-shot evaluation of Qwen2-VL-7B, Qwen2.5-VL-7B, Gemma 3/4 models, and GPT-4o mini shows cognitive asymmetry: semantic understanding reaches a high ceiling, whereas factual recall and creative synthesis remain substantially weaker. The supplied paper text does not provide one aggregate score.
- A stratified 969-item audit flagged 15 potentially incorrect samples; human verification confirmed these cases, yielding a reported 98.45% quality rate. English performance is generally higher than Arabic, exposing a cross-lingual multimodal reasoning gap.
Significance
The benchmark changes evaluation from “how many questions are correct?” to “which cognitive processes are reliable?” This helps researchers diagnose failures that aggregate scores conceal, including recall, transfer, analysis, judgment, and synthesis. Its English-Arabic design also makes linguistic equity and cultural inclusivity first-class evaluation targets, with implications for education, public services, and globally deployed multimodal AI.
Technical Contribution
The main contribution is an extensible cognitive data-engineering and evaluation protocol rather than a new VLM architecture. Its components are a hierarchical task tree, authentic web images, four-choice distractors with a trap option, Gemini 2.5 Pro generation, Arabic translation, Gemini 3 Pro auditing, and human checks. Regex-based Answer Extraction (RAE) measures explicit answer behavior, while Likelihood-based Scoring (LBS) evaluates conditional choice probabilities.
Novelty
The authors present BloomBench as the first multimodal benchmark explicitly grounded in Bloom’s Taxonomy while jointly supporting English and Arabic with real-world imagery. Unlike MMMU, MMT-Bench, and VLM2-Bench, which primarily aggregate heterogeneous tasks, it organizes evaluation around cognitive depth, diagnostic profiles, bilingual fidelity, and cultural coverage.
Limitations
- The dataset is imbalanced: Remember has 2,948 items, whereas Apply has only 499. Layer averages should therefore not be interpreted as equally sampled estimates.
- Create is operationalized as selecting the best candidate in an MCQ, or discriminative creativity. This does not measure open-ended generation, sustained planning, or authentic creative production.
- Quality assurance covers a stratified 969-item subset and depends partly on Gemini 3 Pro; the supplied results emphasize trends rather than complete scores, confidence intervals, or significance tests.
Future Work
Future work should add Arabic dialects, more languages, balanced task sampling, open-ended generation, process supervision, calibration, and human cognitive baselines. Researchers should report per-model, per-level confidence intervals, test contamination, visual evidence citations, and whether Bloom’s hierarchy is genuinely cumulative for VLMs.
AI Executive Summary
Vision-language models can now interpret images and answer questions, yet a strong aggregate score does not establish human-like multimodal reasoning. Existing benchmarks such as MMMU, MMT-Bench, and VLM2-Bench often combine disconnected capabilities, allowing shortcut strategies and offering limited diagnosis of whether a failure reflects perception, memory, transfer, judgment, or creativity.
Almieyar-Oryx-BloomBench proposes a cognitive alternative. It organizes 106 visual task types around Bloom’s six levels—Remember, Understand, Apply, Analyze, Evaluate, and Create—and supports both English and Arabic. Gemini 2.5 Pro proposes culturally diverse scenarios and generates visually grounded VQA; an instruction-tuned model converts them into four-option questions and Modern Standard Arabic. Gemini 3 Pro and human reviewers audit a stratified subset. The resulting resource contains 7,747 bilingual items and reports a 98.45% quality rate.
Zero-shot tests of Qwen2-VL-7B, Qwen2.5-VL-7B, Gemma 3/4, and GPT-4o mini reveal an uneven cognitive profile: semantic understanding is comparatively strong, while factual recall and creative synthesis are substantially weaker. English also outperforms Arabic overall. BloomBench’s central value is therefore diagnostic rather than architectural: it provides a structured capability profile and argues that future VLM progress must track cognitive layers, confidence, and linguistic fairness—not only headline accuracy.
Deep Analysis
Background
Transformers, large-scale corpora, and LLM scaling drove VLM progress. CLIP established influential image-text alignment, while GPT-4o mini, Qwen2.5-VL, and Gemma models support VQA, document understanding, and reasoning. Yet broad evaluations including MMMU, MMT-Bench, and VLM2-Bench remain largely task collections, making continuous cognitive depth difficult to measure.
Core Problem
The problem is to diagnose multimodal reasoning across realistic images rather than collapse heterogeneous abilities into one mean score. A model may understand scene relations but fail factual recall, transfer, safety judgment, or creative selection. English-centric resources further conceal failures caused by language imbalance and cultural mismatch.
Innovation
The work contributes a Bloom-based hierarchy; VLM-specific grounding, spatial and compositional reasoning, logic, science, safety, and creative-choice tasks; authentic web imagery; parallel English-Arabic evaluation; and dual RAE/LBS scoring. These choices differ from benchmarks that mainly increase task breadth, expert difficulty, or dataset scale without an explicit cognitive scaffold.
Methodology
- �� Gemini 2.5 Pro receives a taxonomy path, target skill, and domain, then proposes Western, MENA, and Arabic scenarios plus image-search keywords.
- �� Real web images are sourced and paired with open-ended VQA designed to be answerable from visual evidence.
- �� An instruction-tuned test creator produces the correct option, three plausible distractors, and one trap distractor, followed by Modern Standard Arabic translation.
- �� An LLM filter removes invalid items; 969 examples are stratified across 106 leaves, with at least four per leaf, then audited by Gemini 3 Pro and humans.
- �� Eight VLM configurations run zero-shot at temperature 0. RAE parses A/B/C/D from generated text; LBS compares conditional log-probabilities of answer choices.
Experiments
Models are Qwen2-VL-7B, Qwen2.5-VL-7B, Gemma 3-4B/12B/27B, Gemma 4-26B-A4B/31B, and GPT-4o mini. The 7,747-item bilingual corpus is analyzed by six Bloom levels, language, model scale, and evaluator. English and Arabic are tested with both Regex-based Answer Extraction and Likelihood-based Scoring. The provided text does not specify additional ablations or human-performance baselines.
Results
The principal finding is cognitive unevenness: Understand-related semantic competence reaches a high ceiling, while Remember and Create are more fragile. English generally exceeds Arabic. Of 969 audited items, 15 were confirmed erroneous, corresponding to 98.45% reported quality. Because the supplied paper excerpt lacks the figure’s numerical scores, exact model rankings, effect sizes, and significance cannot be reconstructed responsibly.
Applications
Educators can use layer-specific results to diagnose AI tutors and visual learning systems. Arabic public-service or healthcare interfaces can test bilingual visual comprehension and safety. Model developers can use the 106 task types for release regression tests, data balancing, alignment, and hallucination evaluation, provided that open-ended and domain-expert checks are added before deployment.
Limitations & Outlook
Unequal layer sizes and MCQ formatting constrain interpretation, especially for Create. Semi-automated generation, translation, and model-based judging may share biases. Auditing 969 items is statistically useful but not equivalent to exhaustive review. The current study also lacks complete score tables, confidence intervals, contamination analysis, and human baselines. Future versions should add dialects, generative tasks, process traces, and calibration.
Plain Language Accessible to non-experts
Imagine a school for a robot that can look at pictures. A normal quiz might ask only, “Is there a cat?” That checks recognition, like testing whether a student can read one word. BloomBench runs a much fuller exam. First the robot identifies objects. Then it explains how they relate, applies a learned rule to a new picture, analyzes what is happening, judges whether an answer is safe or sensible, and finally chooses the best ending for a creative story.
Teachers build the exam from real photographs, write parallel English and Arabic questions, and add tempting wrong answers. The robot is checked in two ways: what answer it actually says, and which answer it assigns the highest internal probability. This is like checking both a student’s written answer and the confidence behind it.
The result is a report card, not merely a rank. The robots understand scenes fairly well, but often struggle to recall facts, combine ideas creatively, and perform equally well in Arabic. That tells developers exactly which “school subjects” need more practice.
ELI14 Explained like you're 14
Picture a game where an AI has to answer questions about photos. Level one asks, “Is there a bicycle?” Level two asks what the people and objects are doing together. Level three makes it use a math or science rule. Later levels ask it to spot hidden relationships, catch a misleading answer, and choose the best ending for a story. Pretty different from simply guessing what is in a picture, right?
That is BloomBench. Researchers created 7,747 English-Arabic picture questions covering 106 kinds of tasks. Some wrong choices are deliberately sneaky, so the AI cannot win just by recognizing one obvious word. They also check both the answer the AI writes and the choice it seems most confident about.
The surprising part is that AI can be good at understanding the general meaning of a scene but weaker at remembering exact facts or combining ideas into something creative. Arabic results are usually lower than English results too. Maybe the training material contains more English, or cultural clues do not transfer perfectly.
Think of it like a game character with a strong “scene understanding” stat but weaker “memory” and “creativity” stats. BloomBench helps researchers see those stats instead of hiding everything inside one score. Then training can target the real weak points!
Glossary
Vision-Language Model
A model that processes images and language together. Technically, it aligns visual evidence with language representations to answer or generate text.
BloomBench evaluates Qwen, Gemma, and GPT-4o mini systems.
Bloom’s Taxonomy
An educational hierarchy of cognition from Remember to Create. It structures tasks by increasing reasoning complexity.
The benchmark uses six levels and 106 leaf task types.
Regex-based Answer Extraction (RAE)
A parser that identifies A, B, C, or D in free-form output. Invalid formatting is counted as an incorrect choice.
It measures practical answer generation and instruction following.
Likelihood-based Scoring (LBS)
A method that compares each option’s conditional log-probability given the image and question. It reduces dependence on output formatting.
LBS is paired with RAE to compare confidence and accuracy.
LLM-as-a-judge
Using a language model to assess item validity, answer quality, or translation. It scales review but can introduce evaluator bias.
Gemini 3 Pro audited the 969-item representative subset.
Open Questions Unanswered questions from this research
- 1 It remains unknown whether the six levels form a genuinely cumulative hierarchy for VLMs. Human baselines, learning curves, and staged-prompt experiments are needed.
- 2 The source of the English-Arabic gap—training imbalance, translation, or cultural familiarity—is unresolved. Dialectal and culturally matched datasets are required.
- 3 RAE and LBS may measure different abilities. Calibration and open-ended generation studies should connect confidence, explanations, and creative behavior.
Applications
Immediate Applications
VLM regression testing
Model teams can test every release by Bloom level, language, and task family, detecting regressions in Arabic, recall, or creation that a single overall score would hide.
Bilingual educational assistants
Schools can evaluate whether an AI tutor correctly interprets images, charts, science scenes, and safety situations in English and Arabic, routing failed items to human review.
Long-term Vision
Cognitively aligned multilingual AI
A long-term direction is to use cognitive profiles for training-data allocation, reward modeling, and cross-cultural safety evaluation, producing more equitable and interpretable multimodal systems.
Abstract
Despite the rapid progress of Vision-Language Models (VLMs), the field lacks benchmarks that rigorously diagnose their true reasoning abilities and chart meaningful progress toward human-like multimodal intelligence. Most existing evaluations focus on piecemeal or disconnected tasks, obscuring critical cognitive weaknesses and providing little insight for targeted improvement. To address this gap, we introduce BloomBench, part of the Almieyar benchmarking series, the first cognitively human-grounded, bilingual (English-Arabic) multimodal benchmark for VLMs. Grounded in Bloom's Taxonomy, BloomBench systematically evaluates six levels of cognition (Remember, Understand, Apply, Analyze, Evaluate, Create) through carefully designed image-question-answer tasks. Built with a semi-automated pipeline and validated through a stratified hybrid quality assurance protocol, it ensures scalability, cultural inclusivity, and linguistic fidelity. Leveraging this framework, we conduct a comprehensive study of state-of-the-art VLMs to diagnose their cognitive profiles. Our analysis reveals a sharp cognitive asymmetry: while state-of-the-art models achieve strong performance ceilings in semantic understanding, they struggle substantially with factual recall and creative synthesis. This demonstrates that current general multimodal proficiency masks deeper limitations in specific cognitive layers. Furthermore, our study highlights a critical performance gap between Arabic and English, exposing limitations in current cross-lingual multimodal reasoning. These findings establish a foundation for developing more cognitively aligned and inclusive VLMs. The benchmark framework and dataset is available at: https://github.com/qcri/Almieyar-Oryx-BloomBench.