KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Models

The evolution of multimodal generative models, exemplified by OpenAI’s DALL·E, Stable Diffusion, and Google’s Imagen, has led to remarkable progress in visual quality and diversity. These models leverage large-scale pretraining on image-text pairs, enabling impressive zero-shot capabilities. Despite this, their reasoning abilities, especially in scientific, social, and procedural domains, remain underdeveloped. Existing benchmarks such as EditBench, EMU-Edit, and RISEBench primarily evaluate task-specific performance or visual fidelity, lacking a structured assessment of cognitive reasoning. Recent efforts like IntelligentBench and SmartEdit have begun exploring reasoning dimensions but lack a formal taxonomy rooted in cognitive science. KRIS-Bench advances this by integrating an educational framework, providing a systematic and scalable approach to evaluate reasoning in image editing.

Current models excel at producing visually appealing edits but falter when required to incorporate domain knowledge, perform multi-step reasoning, or apply procedural logic. This gap limits their utility in scientific and educational contexts, where understanding and applying knowledge is critical. The absence of a structured, cognitively grounded evaluation framework hampers targeted improvements. Moreover, existing metrics often fail to capture whether generated edits are plausible within real-world knowledge constraints, risking superficial performance improvements without genuine understanding.

KRIS-Bench’s core innovations include: 1) adopting an educational taxonomy dividing knowledge into factual, conceptual, and procedural types, providing a theoretical basis for reasoning assessment; 2) designing 22 diverse tasks aligned with 7 reasoning dimensions, enabling detailed diagnosis of models’ strengths and weaknesses; 3) introducing a Knowledge Plausibility metric, supported by domain-specific hints and human calibration, to evaluate the real-world consistency of generated edits; 4) creating a large, annotated dataset that supports comprehensive, multi-model evaluation, fostering research on knowledge reasoning in multimodal systems.

• �� Define three knowledge types based on educational taxonomy: factual (observable properties), conceptual (principles and generalizations), procedural (multi-step reasoning).
• �� Design 22 tasks covering attributes, spatial relations, scientific reasoning, social context, and logical operations, each mapped to specific reasoning dimensions.
• �� Collect images from internet, generate synthetic data, and augment prompts with ChatGPT, ensuring diversity and realism.
• �� Annotate data with expert input, including domain-specific knowledge hints, to guide evaluation.
• �� Develop a multi-dimensional evaluation protocol: visual consistency, quality, instruction adherence, and knowledge plausibility.
• �� Use GPT-4o as the automatic evaluator, calibrate with human judgments, and analyze model performance across knowledge types and reasoning complexity.

The evaluation involves 10 models, including GPT-4o, Gemini 2.0, Doubao, and open-source variants like OmniGen and BAGEL. The dataset comprises 1,267 instances across diverse tasks. Metrics include visual consistency, quality, instruction following, and knowledge plausibility, with GPT-4o providing automated scores calibrated via human annotations. Experiments analyze performance across knowledge types, revealing significant gaps in reasoning, especially in procedural and scientific domains. Ablation studies validate the effectiveness of knowledge hints and task design, while cross-model comparisons highlight current limitations and areas for improvement.

Models perform well in visual metrics (average >80), but in knowledge reasoning, scores drop to around 37.15, indicating poor understanding of domain knowledge. GPT-4o leads with 53.36 in knowledge plausibility, yet still exhibits notable deficiencies in complex, multi-step, or domain-specific tasks. Fact-based tasks score above 80, but procedural tasks like instruction decomposition fall below 30, exposing a critical weakness in multi-step reasoning. These findings underscore the urgent need for models to better integrate domain knowledge and reasoning capabilities.

This benchmark enables precise evaluation of models for applications requiring deep understanding, such as scientific visualization, educational content creation, and industrial design. It supports the development of models capable of reasoning about physical laws, social contexts, and procedural steps, facilitating more intelligent and trustworthy AI systems in real-world scenarios. The insights gained can guide targeted model improvements, fostering AI that not only generates visually appealing images but also understands and applies domain knowledge effectively.

The evaluation relies on automated metrics, which may not fully capture nuanced reasoning or domain-specific correctness, necessitating further human validation. Tasks are currently focused on scientific and social reasoning, with less emphasis on multi-modal, dynamic, or real-time knowledge applications. Models still struggle with complex multi-step reasoning and cross-domain generalization, indicating the need for integrating knowledge graphs, reasoning modules, and continual learning mechanisms for future progress.