CoRe: A Comprehensive Framework for Cross-Image Comparative Reasoning in Vision-Language Models

Recent progress in vision-language models (VLMs) such as BLIP, Flamingo, and LLaVA has enabled impressive performance on perception and language understanding tasks. However, these models predominantly excel in single-image perception or broad semantic tasks like captioning and retrieval. The challenge of fine-grained, multi-image relational reasoning—such as counting objects across images, comparing depths, or assessing spatial relations—remains largely unaddressed. Existing benchmarks like MME and MuirBench focus on semantic integration but lack the granularity for detailed relational reasoning. Manual annotation efforts for such tasks are costly and unreliable, while pseudo-labeling introduces errors. Consequently, there is a pressing need for scalable, verifiable datasets and training strategies that explicitly target structured reasoning, which this work aims to fulfill.

Despite advances, current VLMs struggle with tasks requiring precise comparison of attributes across related images. The core issues include the absence of high-quality, automatically generated datasets with verifiable labels, and the lack of training mechanisms that enforce intermediate reasoning correctness. This results in models producing answers that may be correct superficially but are based on flawed reasoning chains, undermining trustworthiness. Addressing these bottlenecks is critical for deploying models in real-world applications like navigation, scene understanding, and surveillance, where accuracy and explainability are paramount.

The paper introduces three key innovations: 1) CoRe-20K, an automatically constructed, high-quality triplet dataset derived from structured visual metadata, covering four relation types; 2) TriSR, a structured reward framework that decomposes reasoning into attribute grounding, judgment alignment, and triplet consistency, providing verifiable intermediate supervision; 3) CoRe-Bench, a specialized benchmark for evaluating fine-grained cross-image reasoning. These components collectively enable models to learn truthful, consistent, and interpretable relational reasoning, surpassing prior methods that rely solely on answer correctness or coarse supervision.

• �� Data Construction: Extract task-specific metrics (count, depth, distance, relations) from structured metadata using dedicated extractors; filter triplets via validity constraints; generate comparison questions with templates and randomized options. • Reward Design: Implement TriSR to evaluate attribute grounding (matching model estimates with metadata), judgment alignment (consistency between reasoning trace and ground truth), and triplet coherence (logical consistency among comparisons). • Training: Use GRPO to optimize model policies based on structured rewards, reinforcing intermediate correctness. • Evaluation: Test on CoRe-Bench with balanced triplet sets, measuring partial accuracy, relation judgment correctness, and reasoning consistency. • Ablation: Validate the impact of each reward component and data quality on overall performance.

The experiments utilize the CoRe-20K dataset, split into training and evaluation subsets, with models like Qwen3-VL and GPT-5.2 as baselines. Metrics include partial accuracy on relation tasks, attribute localization precision, and reasoning chain correctness. Ablation studies compare models trained with and without TriSR rewards, demonstrating significant performance gains. The models are also evaluated on standard benchmarks to confirm that the structured approach does not impair general multimodal capabilities. Hyperparameters are tuned for optimal reward weighting, and robustness is tested across relation types and scene complexities.

Models trained with CoRe and TriSR outperform baselines by 28.2% in partial accuracy on CoRe-Bench. They show improved attribute grounding (up to 15% increase), higher consistency in relation judgments, and better global coherence. Ablation results confirm that structured rewards significantly enhance reasoning fidelity. The models maintain competitive scores on traditional tasks like VQA and captioning, indicating broad applicability. These findings validate the effectiveness of combining automated data generation with structured reinforcement learning for complex relational reasoning.

This framework can be directly applied to navigation systems, scene understanding in autonomous vehicles, surveillance, and robotic perception, where detailed multi-image reasoning is crucial. It enables systems to verify relational attributes explicitly, improving safety and interpretability. Long-term, the approach could facilitate the development of fully autonomous agents capable of complex multi-step reasoning in dynamic environments, advancing AI's ability to understand and interact with the world more reliably.

The reliance on structured metadata limits applicability to datasets with such annotations; real-world unstructured data remains challenging. Computational complexity increases due to the structured reward calculations, impacting scalability. The current approach may struggle with ambiguous or highly complex scenes where attribute extraction and relation verification are difficult. Future work should focus on reducing dependency on metadata, improving efficiency, and extending to less structured environments.