MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models

Multimodal large language models have recently made significant progress in integrating diverse modalities such as vision, audio, and language. These models aim to emulate human-like multimodal perception by jointly processing multiple sensory inputs, enabling applications like video question answering and audio-visual scene understanding. However, despite these advancements, MLLMs still face the challenge of cross-modal hallucinations, where one modality inappropriately influences another, leading to fabricated outputs.

Cross-modal hallucinations are a core issue faced by MLLMs. Traditional hallucination problems typically involve generating incorrect or fabricated information within a single modality. In contrast, multimodal settings introduce a more subtle yet harmful failure pattern: cross-modal hallucinations, where one modality improperly influences content generation about another. Addressing this challenge requires the model to possess modality appropriateness judgment capability.

The core innovation of the MAD method lies in introducing a modality-adaptive decoding mechanism. First, MAD prompts the model to self-assess modality relevance, extracting modality probabilities. Then, it adaptively weights contrastive decoding branches to focus on task-relevant modality information, suppressing irrelevant interference. This method requires no model retraining and offers a principled extension to existing contrastive decoding methods.

• �� MAD leverages the model's self-assessment capability to extract modality probabilities.
• �� It adaptively weights contrastive decoding branches to focus on task-relevant modality information.
• �� Validated on benchmarks like CMM and AVHBench, significantly reducing cross-modal hallucinations in various audio-visual language models.

Experiments were conducted on CMM and AVHBench benchmarks using models like VideoLLaMA2-AV and Qwen2.5-Omni. Compared to existing contrastive decoding methods, MAD demonstrated significant advantages in reducing cross-modal hallucinations.

On CMM and AVHBench, MAD reduced cross-modal hallucinations in VideoLLaMA2-AV by 7.8% and 2.0%. Qwen2.5-Omni showed improvements of 8.7% and 4.7% on the same datasets. Experiments indicate explicit modality awareness is crucial for robust multimodal reasoning.

MAD can be directly applied to audio-visual language understanding tasks, enhancing model accuracy and reliability. Its training-free nature makes it easy to integrate into existing multimodal large language models.

MAD may still misjudge extremely complex multimodal inputs, especially when modality information is unclear. The method relies on the model's self-assessment capability, which may be limited by the model's inherent constraints. Future research could explore applying MAD in more complex multimodal scenarios or combining it with other methods to further enhance hallucination suppression.