COCOLogic-V2: Identifying Logical Inconsistencies via Truly Hard-Negatives
COCOLogic-V2 evaluates visual inductive reasoning with truly hard negatives, revealing models' struggles on boundary samples.
Key Findings
Methodology
This study introduces COCOLogic-V2, based on MSCOCO images, defining 10 first-order logic rules. Samples are categorized into positives, near-boundary (NB), and far-from-boundary (FB) negatives. A sampling strategy balances the dataset, enabling fine-grained diagnosis. Models evaluated include concept bottleneck models (CBMs), vision-language models (VLMs), and program synthesis. Results show models distinguish positives and FB well but fail on NB, indicating shallow understanding. Experiments highlight challenges in few-shot settings due to perceptual noise and large rule search spaces.
Key Results
- Models achieve over 94% accuracy distinguishing positives from FB samples but only 30-50% on NB samples, exposing limited deep logic understanding.
- In few-shot scenarios (24 samples/rule), CBMs perform poorly with high variance, while VLMs improve to ~70% accuracy, yet still struggle with complex rules.
- Program synthesis approaches reach near 90% under ideal conditions but are limited by perception noise and large concept spaces in practice.
Significance
This work underscores the complexity of real-world visual reasoning, revealing that current models rely on statistical shortcuts rather than genuine rule comprehension. COCOLogic-V2 provides a diagnostic framework to evaluate and improve interpretability, crucial for high-stakes applications like autonomous driving and healthcare. It pushes the field toward models that understand deep logical structures, beyond superficial correlations.
Technical Contribution
The paper introduces a novel approach of subdividing negative samples into near-boundary types, enabling detailed model diagnostics. It combines multi-level logic rule design with a strategic sampling process, facilitating precise evaluation of deep understanding. The comprehensive assessment across CBMs, VLMs, and program synthesis advances the understanding of current limitations and guides future research in explainable AI.
Novelty
This is the first systematic use of boundary negative samples for real-world visual reasoning, providing a nuanced evaluation of models’ deep understanding. Unlike prior datasets focusing on synthetic or simple tasks, COCOLogic-V2 captures the complexity of natural images and logic, setting a new benchmark for interpretability and robustness.
Limitations
- Models still underperform on NB samples, indicating that deep logical understanding remains a challenge, especially in few-shot scenarios.
- Perception noise and large concept spaces hinder program synthesis and VLM performance, necessitating better perception modules.
- The dataset’s logical rules, while diverse, still simplify real-world complexity; future work should include more intricate relations and scene-level reasoning.
Future Work
Future directions include expanding rule complexity, integrating relational reasoning, and incorporating scene context. Combining multi-modal data and reinforcement learning could further enhance model robustness and interpretability, moving toward truly generalizable visual reasoning systems.
AI Executive Summary
As artificial intelligence advances, the ability of models to perform deep, interpretable visual reasoning remains a critical challenge. Existing approaches like concept bottleneck models (CBMs) and neuro-symbolic methods excel in simple tasks but falter in complex, real-world scenarios. Recognizing this gap, Steinmann et al. introduce COCOLogic-V2, a novel dataset built on MSCOCO images, designed to evaluate logical reasoning with fine-grained diagnostic capabilities.
COCOLogic-V2 encompasses 10 first-order logic rules, including object counting, absence, and relational constraints. Samples are categorized into positive variants, near-boundary (NB), and far-from-boundary (FB) negatives, enabling detailed assessment of models’ understanding. The dataset’s sampling strategy ensures balanced representation, facilitating robust evaluation. Experiments with CBMs, vision-language models (like CLIP and DCR), and program synthesis reveal that while models distinguish easy positives and FB negatives effectively, they struggle with NB samples, exposing superficial pattern reliance.
In few-shot settings, the challenge intensifies. CBMs show high variance and limited generalization, whereas VLMs, especially with in-context learning, achieve around 70% accuracy but still fall short on complex rules. Program synthesis approaches perform well under ideal conditions but are hampered by noisy perception modules and large concept spaces. These findings highlight the persistent difficulty of deep logical understanding in real-world images, emphasizing the need for more sophisticated perception and reasoning modules.
Overall, COCOLogic-V2 offers a crucial benchmark for advancing interpretable AI. It reveals that current models rely heavily on statistical shortcuts, underscoring the importance of developing methods that truly grasp deep logical structures. Future work should focus on expanding logical complexity, scene-level reasoning, and multi-modal integration, paving the way for AI systems capable of reliable, explainable reasoning in complex environments.
Deep Analysis
Background
随着深度学习在视觉任务中的广泛应用，模型的可解释性和逻辑推理能力成为研究热点。早期工作如Concept Bottleneck Models（CBMs）和神经符号方法，主要解决单一类别识别或符号推理，但在复杂场景中的表现有限。近年来，COCOLogic等数据集推动了逻辑推理的研究，但多为合成或单标签任务，难以反映真实世界的复杂性。实际应用中，模型需要理解对象关系、计数和缺失等多层次逻辑，现有方法在边界样本和少样本场景下表现不足，亟需更具挑战性和细粒度的评估工具。
Core Problem
核心问题在于当前模型在真实场景中的深层逻辑理解能力不足，尤其在边界样本（即模型难以判断的样本）上表现差强人意。现有数据集多忽略样本的细粒度分类，导致模型只在简单场景下表现良好，无法应对复杂推理任务。少样本学习更是加剧了这一问题，模型难以捕获规则的细节，导致推理不可靠。这些限制严重影响模型在实际应用中的可信度和鲁棒性。
Innovation
本研究的创新点在于引入边界负样本作为细粒度诊断工具，系统性划分正负样本，揭示模型在深层逻辑理解上的短板。通过多阶逻辑规则设计，结合样本细分策略，构建了COCOLogic-V2数据集，显著提升了推理任务的难度和诊断能力。采用多模型评估，包括CBMs、VLMs（如CLIP、DCR）和程序合成，全面分析模型在不同逻辑场景中的表现差异，推动了可解释性和鲁棒性研究的结合。
Methodology
- �� 设计10个基于MSCOCO的第一阶逻辑规则，定义正样本、边界负样本（NB）和远离边界负样本（FB）。
- �� 通过Disjunctive Normal Form（DNF）将规则拆解为多种正样本变体，每个变体代表一种满足规则的方式。
- �� 采样策略：从MSCOCO中随机抽取样本，确保每个正样本变体至少有1000个样本，边界负样本至少500个，剩余样本补充FB样本，保持数据分布接近原始。
- �� 评估模型：利用CBMs、VLMs（如CLIP、DCR）和程序合成（Gemma-4-31B-it）进行多角度评估，指标包括平衡准确率（B-Acc）和边界样本（NB）和正样本的单独准确率。
- �� 细粒度诊断：分析模型在正样本、NB和FB上的表现差异，揭示模型在深层逻辑理解上的不足。
Experiments
实验在COCOLogic-V2和少样本版本COCOLogic-V2-FS上进行。采用多种模型：包括基于概念的CBMs、预训练VLM（如CLIP）、程序合成方法和大规模语言模型（如GPT-5.5）。指标主要为平衡准确率（B-Acc）及边界样本（NB）和正样本的单独准确率。通过不同感知模块（如Mask-RCNN、手工标注）和少样本训练，分析模型在不同逻辑规则上的表现差异。还进行了消融实验，验证样本细分策略的有效性。
Results
模型在整体上对正样本和FB样本表现良好（准确率超过94%），但在NB样本上表现极差（30-50%），显示深层逻辑理解不足。少样本条件下，CBMs表现不稳定，VLMs通过In-Context Learning提升显著，达70%以上，但对复杂规则仍有困难。程序合成在理想状态下接近90%，但实际受限于感知噪声和搜索空间。多模型结果共同揭示了模型在边界样本上的普遍短板，强调了细粒度样本分类的重要性。
Applications
该数据集和评估框架适用于开发和测试具备深层逻辑理解能力的视觉推理模型，特别是在自动驾驶、医疗影像和智能监控等高风险场景中。模型需结合对象检测、关系推理和少样本学习，才能实现可靠的逻辑推断。未来，结合多模态信息和强化学习，有望突破现有瓶颈，推动AI系统的可信性和解释性。
Limitations & Outlook
模型在边界负样本上的表现仍不理想，说明深层逻辑理解仍需突破。感知噪声和大规模概念空间限制了推理能力，未来需改进感知模块和推理机制。数据集虽丰富，但规则复杂度和场景多样性有限，未来应扩展逻辑结构和场景复杂度，提升模型泛化能力。
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂里有很多不同的机器和工人，每个人都负责不同的任务。有时候，工厂需要判断某个产品是否符合特定的标准，比如是否有两个按钮，或者没有某个零件。传统的工人（模型）可能只会看一眼就判断，但实际上，理解这些规则需要他们知道每个零件的数量和位置。COCOLogic-V2就像是给工人们提供了一份详细的说明书，告诉他们在不同情况下应该怎么判断。通过不断练习，工人们可以学会更好地理解这些规则，但有时候他们还是会在边界情况下犯错，比如只差一个零件就符合规则。这项研究就是在测试工人们是否真正理解规则，帮助他们变得更聪明、更可靠。
ELI14 Explained like you're 14
想象你在学校里参加一个游戏，老师给你一些规则，比如“如果你有两个苹果和一个香蕉，你就赢了”。但有时候，你可能只差一点点，比如只有一个苹果或者两个香蕉。这个游戏的难点在于，你要真正理解这些规则，而不是只看表面。科学家们做了一个类似的游戏，让电脑学习这些规则，看看它们是不是真的懂得。为了测试，科学家们设计了一些特别的图片，有的符合规则，有的只差一点点（边界样本），还有一些完全不符合（远离边界）。他们发现，电脑在判断简单的图片时还可以，但在那些只差一点点的图片上就糊涂了，就像你在游戏中只差一点点就赢了。这个研究帮助我们知道，电脑还需要学习得更聪明，才能像人一样真正理解规则，不会被表面现象骗到。
Abstract
While interpretable models such as concept bottleneck models (CBMs) and program synthesis methods enable verification of model decisions, their evaluation is typically limited to simple tasks, leaving complex reasoning on real-world images largely unexplored. We introduce COCOLogic-V2, an object-centric dataset for visual inductive reasoning on real-world images covering a broad subset of first-order logic. By categorizing samples into positive variants, near-boundary (NB), and far-from-boundary (FB) negatives, COCOLogic-V2 enables fine-grained diagnosis of model accountability. Our evaluations show that models tend to separate positive and FB samples well but fail on NB samples, while perceptual noise and large rule-induced search spaces pose additional challenges in few-shot settings. Together, these results highlight that visual inductive reasoning remains an open challenge and COCOLogic-V2 provides a concrete foundation for advancing methods in this direction.