Adaptive Perturbation Selection for Contrastive Audio Decoding
Proposes adaptive perturbation selection via structured audio transformations, boosting contrastive decoding accuracy to 81.4% on temporal tasks.
Key Findings
Methodology
This work introduces a library of 105 targeted audio perturbations across temporal, spectral, frequency, and amplitude domains. A lightweight neural selector is trained on model hidden states to dynamically route each input to the most effective negative branch, optimizing contrastive decoding performance. The approach incorporates a binary yes/no prompt constraint to calibrate model bias, significantly improving accuracy. Experiments on Qwen2-Audio-7B-Instruct and Audio Flamingo 3 across tasks like existence, temporal order, and attribute detection demonstrate that task-dependent perturbations, such as reversing audio for order tasks, yield substantial gains. The adaptive selector further enhances performance by selecting the optimal perturbation per sample, achieving +4.3% on existence tasks, with remaining oracle headroom of 9.5%.
Key Results
- The binary yes/no prompt reduces affirmative bias, increasing existence task accuracy by 11%.
- Task-specific perturbations like reversing audio improve temporal order accuracy from 74.7% to 81.4%.
- Adaptive routing with a hidden-state trained selector boosts existence task accuracy by 4.3%, nearing oracle performance with 9.5% headroom.
Significance
This research advances contrastive decoding by integrating structured, task-aware audio perturbations, addressing the core issue of hallucinations in large audio-language models. It demonstrates that dynamic, learned routing of negative examples significantly enhances model robustness and fidelity, paving the way for more reliable multimodal AI systems. The framework offers a new paradigm for inference-time self-correction, with broad implications for speech recognition, audio understanding, and multimodal content generation.
Technical Contribution
Key innovations include a comprehensive audio perturbation library covering diverse transformation types, a lightweight neural selector trained on internal model states for dynamic negative branch routing, and the integration of prompt calibration to reduce bias. The approach moves beyond static perturbation strategies, enabling task-dependent, sample-specific contrastive examples. This combination yields substantial performance improvements without additional training, representing a significant step forward in inference-time model correction.
Novelty
This is the first systematic evaluation of structured, task-dependent audio perturbations within contrastive decoding. Unlike prior work relying on generic noise or masking, this method designs targeted transformations aligned with specific failure modes, coupled with a learned routing mechanism. The adaptive framework demonstrates superior gains across multiple tasks, establishing a new standard for inference-time model calibration and hallucination mitigation.
Limitations
- The perturbation library, while extensive, cannot cover all possible audio failure modes, limiting effectiveness in highly complex or novel scenarios.
- The selector’s training relies on limited labeled data, which may affect generalization to unseen or rare cases.
- Increased inference complexity and computational overhead may hinder real-time deployment in resource-constrained environments.
Future Work
Future directions include expanding the perturbation library to include more complex, real-world audio distortions, developing self-supervised training for the selector to improve generalization, and integrating fine-tuning strategies to internalize perturbation utility within the model representations. Exploring sequence-level divergence for open-ended generation and captioning tasks also presents promising avenues for extending adaptive contrastive decoding.
AI Executive Summary
Large audio-language models have revolutionized multimodal understanding but are plagued by hallucinations—generating plausible yet false content. Existing mitigation strategies like contrastive decoding (CD) rely on simple negative examples such as masking or noise, which often fail to address structured audio failure modes. This paper introduces a novel framework that leverages a structured library of 105 targeted audio perturbations, spanning temporal, spectral, frequency, and amplitude domains. These perturbations are designed to target specific model failure modes, such as temporal order errors or false existence claims.
A key innovation is the training of a lightweight neural selector, which, based on the model’s internal hidden states, dynamically routes each input to the most effective negative branch. This adaptive approach allows the model to tailor its contrastive examples to the task and sample, significantly improving performance. Experiments on two large models across multiple tasks demonstrate that task-dependent perturbations like reversing audio for temporal order tasks can boost accuracy from 74.7% to 81.4%. Additionally, the use of a binary yes/no prompt constraint reduces affirmative bias, further enhancing accuracy.
The results show that combining structured, task-aware perturbations with adaptive routing yields substantial gains, reducing hallucinations and improving model reliability. This work opens new avenues for inference-time self-correction in multimodal models, with broad implications for speech recognition, audio understanding, and content generation. Despite these advances, challenges remain in expanding the perturbation library, improving selector generalization, and reducing computational costs for real-time applications. Future efforts will focus on these areas, aiming to develop more robust, scalable, and intelligent audio-language systems.
Deep Analysis
Background
随着大型音频-语言模型（LALMs）在语音识别、音频理解等任务中的广泛应用，虚假幻觉问题逐渐凸显。早期研究如SALMONN、Audio Flamingo等通过改进模型结构和训练策略提升性能，但在推理阶段仍面临偏差和虚假生成的挑战。对比解码（CD）作为一种推理时校正技术，通过比较专家模型与扰动负样本的输出差异，有效缓解虚假幻觉，已在多个任务中展现潜力。此前多依赖噪声或掩码扰动，缺乏结构化、目标导向扰动的系统探索。近年来，结构化扰动如反转、频移等被逐步引入，但缺乏任务依赖的优化机制，限制了其效果。本文在此基础上，提出多样化扰动库和自适应路由机制，旨在实现更精细的模型校正。
Core Problem
尽管对比解码在减少虚假幻觉方面取得一定成效，但其多依赖静态、通用的扰动策略，难以应对不同任务和样本的多样性。模型在某些场景下依然会被强烈的语言偏好所误导，导致虚假确认。如何设计结构化、目标导向的扰动，结合模型内部表示实现动态路由，成为提升解码效果的关键。现有方法缺乏系统的扰动库和任务依赖的扰动优化机制，限制了其在复杂多变场景中的应用潜力。
Innovation
本文的核心创新在于：1）构建105种跨越时间、频谱、频率和振幅的结构化扰动库，覆盖多样化音频变换；2）引入二元yes/no约束，调节模型偏置，显著提升存在性任务的准确率；3）训练基于模型隐藏状态的轻量级扰动选择器，实现对不同任务和样本的动态扰动路由。这一框架突破了以往静态扰动设计的局限，结合任务依赖性和模型内部表示，显著改善虚假幻觉问题。
Methodology
- �� 构建扰动库：设计涵盖105种扰动，分为时间、频谱、频率和振幅四大类，利用librosa和SciPy实现具体变换。
- �� 设计二元yes/no约束：在提示中加入明确的回答限制，校正模型偏差。
- �� 训练扰动选择器：采集模型隐藏状态，训练多标签二元分类模型预测扰动效果。
- �� 动态扰动路由：在推理时，根据选择器输出选择最优扰动，结合对比解码实现模型校正。
- �� 评估指标：在多个任务（存在性、时间顺序、属性）上衡量准确率，比较不同扰动策略和选择器配置的效果。
Experiments
采用两个大模型（Qwen2-Audio-7B-Instruct和Audio Flamingo 3）在四个任务（AH Existence、AH Order、AH Attribute、Clotho-AQA）上进行评估。通过五个不同的扰动组合，比较静态固定扰动与动态选择器的性能差异。调节α参数，分析扰动对模型输出的影响。采用五折交叉验证，确保结果稳健。重点实验包括扰动库规模、选择器结构、输入特征配置和正则化策略的 Ablation。
Results
引入二元yes/no约束后，模型在存在性任务中的准确率提升11%，在时间顺序任务中，反转音频扰动提升7%，频移扰动提升4.4%。动态扰动选择器在存在性任务中再提升4.3%，剩余oracle空间达9.5%。扰动库规模扩大至60个候选时，oracle性能达86.4%，但选择器最优性能在4个候选时（76.7%）已接近极限。调节α参数，发现α=1.0为最佳平衡点。输入特征分析显示，最后一层隐藏状态和多层拼接信息最具代表性。正则化策略（标签平滑、特征噪声）有效防止过拟合，提升模型泛化能力。
Applications
该方法适用于多模态语音识别、音频内容理解和虚假信息检测等场景。通过结构化扰动和自适应路由，提升模型在复杂环境中的鲁棒性，减少虚假幻觉。未来可结合微调和多模态训练，构建更智能的自我校正系统，推动语音交互和内容生成的行业应用。
Limitations & Outlook
扰动库虽丰富，但仍未覆盖所有潜在失效场景，复杂音频环境中效果有限。选择器训练依赖有限样本，泛化能力受限。推理过程增加计算复杂度，影响实时性。未来需扩展扰动类型，优化模型结构，提升效率。
Plain Language Accessible to non-experts
想象你在一家工厂里，工厂每天生产各种产品。有时候，产品出现瑕疵，工厂需要找出问题所在。传统方法就像用一个简单的放大镜，只能看一部分问题，比如裂缝或变色，但不能看整体。现在，工厂引入了多种工具，比如热成像、声音检测、振动分析，专门针对不同问题设计。工厂还配备了一个智能助手，能根据不同的产品和问题，自动选择最合适的工具。这样一来，工厂可以更快、更准确地找到问题，修复产品。本文的技术就像这个工厂的智能助手，能根据音频的不同特性，自动选择最佳的扰动策略，帮助模型更好地理解和校正虚假信息。
ELI14 Explained like you're 14
想象你在玩一款超级复杂的游戏，里面的角色会说话、做动作，但有时候会出现奇怪的错误，比如角色突然说出不存在的台词或者做错动作。科学家们也遇到类似的问题，叫做“虚假幻觉”，就是模型自己编造了不存在的内容。为了让模型更聪明，研究人员设计了一种方法，就像给游戏角色加上“调节器”，让它在说话前先用不同的“滤镜”检查一下内容是不是合理。这些“滤镜”可以是让声音变得更模糊、扭曲或者反转，就像用不同的滤镜拍照一样。然后，模型会根据这些“滤镜”判断哪些内容是真实的，哪些可能是虚假的。最厉害的是，他们还让这个“调节器”自己学习，知道什么时候用哪个滤镜最合适。这样一来，模型就能更聪明地避免虚假信息，变得更可靠，就像你在游戏中变成了一个超级侦探，能快速找出真假。
Abstract
Large audio-language models (LALMs) frequently hallucinate by overriding acoustic evidence with language priors. While contrastive decoding (CD) offers training-free mitigation, existing methods rely on blunt perturbations like masking or noise, leaving structured audio transformations unexplored. We explore this design space by evaluating a diverse library of targeted audio perturbations and adaptively selecting the optimal negative branch for each task and example. First, we improve upon earlier prompt engineering by showing that a simple binary yes/no constraint reduces the model's tendency to falsely confirm absent audio features. Second, evaluating our library across temporal, spectral, frequency, and amplitude domains reveals that optimal transformations are highly task-dependent; for instance, reversing the audio array disrupts temporal coherence, raising accuracy on the temporal order task from 74.7% to 81.4%. Finally, we trained a light-weight perturbation selector on model hidden states to dynamically route negative branches, yielding an additional +4.3% gain on the existence task.