Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation

Multimodal learning has undergone rapid evolution, from early datasets like Flickr30k and MS COCO to complex reasoning benchmarks such as VQA, MMBench, and MMMU. Generative models like diffusion and GANs have revolutionized image synthesis, enabling high-fidelity visual content creation and editing. Despite these advances, most research focuses on isolated tasks, lacking a unified assessment of models’ ability to integrate understanding and generation. This gap hampers progress in developing truly versatile AI systems capable of complex cross-modal reasoning and synthesis, which are crucial for real-world applications like autonomous driving, content creation, and human-computer interaction.

Current evaluation paradigms treat understanding and generation as separate tasks, neglecting their interdependence. Models often excel in single capabilities but falter when required to perform joint reasoning and synthesis, especially in multi-step, attribute-sensitive scenarios. This fragmentation limits the development of truly intelligent multimodal systems, as it does not reflect the integrated nature of human perception and cognition. The challenge lies in designing a comprehensive, interpretable, and human-aligned evaluation framework that can accurately measure the synergy between understanding and generation, guiding future model improvements.

Unison introduces a multi-dimensional evaluation framework that decomposes model capabilities into four key aspects: internal consistency, understanding-guided generation, generation-guided understanding, and mutual enhancement. It employs a carefully curated dataset of 169 samples covering attribute recognition, spatial reasoning, and iterative self-optimization. The framework integrates algorithms like VQA, text-to-image synthesis (Qwen-Image), and multi-round self-refinement, providing a holistic view of model performance. The Unison-Judge, a fine-tuned multimodal large model, aligns evaluation with human preferences, ensuring reliability and interpretability. This approach surpasses traditional single-metric evaluations, offering a new paradigm for comprehensive assessment.

• �� Construct four evaluation dimensions: internal consistency, understanding-guided generation, generation-guided understanding, and mutual enhancement.
• �� Curate 169 high-quality samples, including attribute detection, spatial reasoning, and multi-step optimization tasks.
• �� Use algorithms like VQA for attribute comprehension, Qwen-Image for image synthesis, and iterative self-refinement for model self-improvement.
• �� Develop Unison-Judge by fine-tuning a multimodal large model on 8,000 labeled samples, aligning evaluation with human preferences.
• �� Implement multi-round interaction loops where models generate, evaluate, and refine outputs, measuring consistency and improvement through metrics like IoU, Jaccard, and output agreement.
• �� Perform extensive experiments across open-source and closed-source models, comparing scores and analyzing failure modes to identify bottlenecks.

The evaluation includes models like Show-o, Janus-pro, D-DiT, Gemini 3 Pro, and GPT-5.2, tested on datasets covering attribute recognition, spatial reasoning, and multi-step refinement. Metrics such as IoU, Jaccard, and preference correlation are used. Experiments assess the impact of multi-round self-optimization, attribute consistency, and human-aligned evaluation. Ablation studies analyze the contribution of each component, revealing that models with multi-round refinement and bias correction achieve higher scores. Results demonstrate that the best models reach an internal consistency of 0.75 and a preference alignment of 0.89, indicating significant progress but also highlighting remaining challenges in complex reasoning and attribute preservation.

Models show a marked improvement in internal consistency, with top scores reaching 0.75, and Unison-Judge's alignment with human preferences surpassing 0.89. Multi-round self-optimization notably enhances performance in attribute-sensitive and multi-step tasks. The experiments reveal that current models still face difficulties in complex reasoning, especially in multi-turn scenarios, emphasizing the need for more sophisticated reasoning modules and better attribute fidelity. These insights guide future research directions, aiming to develop models with stronger integrated capabilities.

The framework can be applied in AI development pipelines for autonomous systems, content creation, and intelligent assistants, providing a comprehensive performance measure. It helps developers identify weaknesses in understanding and generation, guiding targeted improvements. The human-aligned evaluation ensures models meet user expectations, fostering trust and usability in real-world applications. Additionally, the benchmark supports ongoing research in multi-task learning, multi-turn reasoning, and cross-modal understanding, accelerating progress toward more human-like AI systems.

Current evaluation samples are mostly static, limiting insights into models' dynamic interaction capabilities. The Unison-Judge, while aligned with human preferences, still exhibits biases and requires further refinement. Models struggle with highly complex, multi-turn reasoning tasks, especially in extreme scenarios, indicating the need for more advanced reasoning and adaptive mechanisms. Future work should address these limitations by incorporating real-time interaction data, expanding sample diversity, and enhancing bias correction methods.