Rethinking CD: A Reproducibility Study and Extension on the Ineffectiveness of Contrastive Decoding at Mitigating Object Hallucinations in MLLMs
Reproduction and extension of contrastive decoding (CD) reveal its effects are mainly due to distribution shifts, not true hallucination mitigation, across multiple models and datasets.
Key Findings
Methodology
The study reproduces Yin et al. (2026)βs core claims: CD causes a unidirectional bias towards 'Yes' responses, and APC reduces sampling to greedy decoding. Experiments on LLaVA-v1.5-7B/13B and Qwen2.5-VL-7B across POPE, MME, CHAIR datasets validate these claims. Additional analyses include logit distribution examination, proxy methods for bias source validation, and layer-wise propagation of hallucination signals, revealing the superficial nature of reported gains.
Key Results
- Across datasets, methods like VCD and SID increase 'Yes' predictions by 8-10%, but similar shifts occur with non-visual controls like PBA and OLM, indicating bias rather than improved visual grounding. For example, on POPE Adversarial split, 'Yes' rate rose from 54.87% to over 62% with CD methods, yet hallucination rates remained unchanged.
- Logit distribution analysis shows no effective suppression of hallucinated tokens; biases are mainly due to linear shifts in output probabilities. Proxy models with added Gaussian noise replicate CD effects, confirming bias origin.
- Layer-wise logit-lens analysis indicates that hallucination signals are linearly decodable at intermediate layers, but CD acts as a uniform translation of the 'Yes'-vs-'No' margin, not mechanism correction, thus failing to address core issues.
Significance
This work critically evaluates the effectiveness of current contrastive decoding techniques, emphasizing that apparent improvements are often superficial and do not enhance true visual grounding. It urges the community to develop more robust, explainable, and visual-aware hallucination mitigation strategies, especially vital for high-stakes applications like autonomous systems and medical diagnostics.
Technical Contribution
The paper systematically reproduces key claims, introduces proxy bias analysis, and applies layer-wise interpretability tools to dissect hallucination propagation. It clarifies that distribution shifts, rather than genuine visual understanding, underlie reported gains, providing a new theoretical perspective and practical guidance for future model design.
Novelty
This is the first comprehensive, multi-model, multi-dataset study to dissect the superficiality of CD gains, combining bias source validation with mechanistic layer analysis. It shifts focus from metric improvements to understanding the underlying biases, offering a new paradigm for hallucination research.
Limitations
- The analysis is limited to specific models and datasets; results may vary with other architectures or real-world scenarios. The proxy models, while insightful, do not capture all complexities of hallucination mechanisms.
- The study emphasizes statistical bias, but visual feature integration and multi-modal fusion strategies remain underexplored. Causal mechanisms behind hallucination persistence need further investigation.
- Computational costs of detailed layer-wise analysis are high, and extending these insights to large-scale deployment requires further research.
Future Work
Future research should focus on integrating visual feature enhancement with multi-modal alignment, developing explainable bias correction mechanisms, and designing training strategies that directly target hallucination roots. Exploring causal models and neuro-inspired architectures may yield more robust solutions, ultimately enabling trustworthy multimodal AI systems.
AI Executive Summary
Multimodal large language models (MLLMs) have revolutionized AI by enabling complex visual and language tasks. However, a persistent challenge remains: object hallucination, where models generate objects not present in the visual input. This issue undermines trust, especially in critical applications like autonomous driving and healthcare.
Contrastive Decoding (CD) emerged as a promising, training-free approach to mitigate hallucinations by contrasting model outputs under different input perturbations. Yet, recent critiques, notably by Yin et al. (2026), question whether CD truly improves visual grounding or merely shifts output distributions. This study rigorously reproduces their experiments across models like LLaVA and Qwen, confirming that CD induces a unidirectional bias toward 'Yes' responses. Importantly, the apparent gains are often matched by non-visual controls, indicating superficial effects.
Further analysis reveals that CD's bias stems from distribution shifts rather than enhanced visual understanding. Layer-wise interpretability shows hallucination signals are already decodable at intermediate layers, and CD acts as a linear translation of the 'Yes'-vs-'No' margin, not a mechanism correction. These findings challenge the efficacy of current contrastive strategies, emphasizing the need for approaches that incorporate visual features more deeply.
Overall, this work underscores that superficial metric improvements do not equate to genuine hallucination mitigation. It calls for a paradigm shift toward more explainable, visual-aware, and causally grounded methods, paving the way for trustworthy multimodal AI systems in high-stakes domains.
Deep Dive
Abstract
Contrastive decoding (CD) has been proposed as a training-free strategy for mitigating object hallucinations in multimodal large language models (MLLMs), with reported gains on benchmarks such as POPE. However, recent work has questioned whether these gains reflect genuine improvements in visual grounding. In this study, we reproduce and extend the findings of "The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs." Specifically, we test the claim that CD induces a unidirectional output distribution shift in discriminative datasets and examine its generalizability across datasets. We also verify that the adaptive plausibility constraint (APC) reduces sampling to greedy search on both discriminative and generative benchmarks. Beyond reproduction, we rigorously study the effects of CD across generative and discriminative datasets. We conduct several experiments that provide additional insights: we analyze the logit distributions induced by different CD strategies on generative datasets, propose a proxy method and compare its performance against CD techniques, and investigate how hallucination signals propagate through each layer of the expert and amateur models. Experimental results across MME, POPE, and CHAIR using LLaVA and Qwen validate the original claims and show that the apparent improvements from CD are often spurious and do not consistently translate into stronger visual grounding for reducing hallucinations. These findings challenge the effectiveness of current contrastive decoding strategies and motivate the development of more reliable approaches for mitigating hallucinations in MLLMs.