CORD: Bridging the Audio-Text Reasoning Gap via Weighted On-policy Cross-modal Distillation
CORD employs weighted on-policy cross-modal self-distillation, using text as internal teacher, to reduce audio-text reasoning gap by over 40%.
Key Findings
Methodology
CORD integrates multi-level alignment by combining token-level importance-weighted reverse KL divergence and sequence-level global reward via judge-guided GRPO. It performs online self-distillation within a single model, aligning audio-conditioned reasoning trajectories with text-conditioned behaviors. The token-level component emphasizes early and semantically critical tokens, while the sequence-level component uses a discriminator to evaluate overall reasoning consistency, optimizing via GRPO. Training on only 80K synthetic samples, the framework significantly narrows the modality gap.
Key Results
- On Qwen2-Audio-7B-Instruct, CORD reduces the audio-text reasoning gap by 41.6%, outperforming traditional distillation methods that only achieve 28.5%.
- On Step-Audio2-mini, the gap shrinks by 44.8%, demonstrating superior cross-modal alignment.
- The method nearly closes the modality gap in several tasks, validating the effectiveness of multi-granularity, on-policy alignment.
Significance
This work addresses the longstanding challenge of modality mismatch in large audio-language models, enabling more accurate and robust reasoning across audio and text. By internalizing the teacher within the same model and dynamically correcting semantic deviations, it reduces data dependency and enhances scalability. The approach offers a new paradigm for efficient multimodal training, with broad implications for speech understanding, intelligent QA, and human-computer interaction, pushing the field toward more unified and capable models.
Technical Contribution
CORD introduces a novel combination of importance-aware token-level reverse KL divergence and judge-based sequence-level reward optimization within an end-to-end framework. It leverages in-model text modality as a self-teacher, avoiding off-policy issues and architecture noise. The method guarantees targeted correction of critical semantic tokens and global trajectory alignment, providing theoretical and empirical improvements over prior approaches like off-policy distillation and static representation alignment. It expands the application of self-distillation to complex multimodal reasoning tasks.
Novelty
This is the first work to implement multi-granularity, on-policy self-distillation for audio-text reasoning within a single model, utilizing importance-weighted reverse KL and judge-guided GRPO. Unlike previous methods relying on external teachers or static alignment, it dynamically corrects reasoning trajectories, achieving significant performance gains and closing the modality gap. This innovative integration of token-level and sequence-level alignment sets a new standard in multimodal model training.
Limitations
- The approach relies heavily on synthetic data, which may not fully capture real-world audio variability, limiting real-world robustness.
- The judge model, though accurate, may introduce biases affecting global reward signals, potentially impacting generalization.
- In extremely complex reasoning scenarios, the current framework may still face challenges, necessitating further integration of richer multimodal features.
Future Work
Future directions include incorporating real-world noisy audio data, refining judge models for unbiased global rewards, and extending the framework to multi-task, multi-modal settings. Combining reinforcement learning techniques could further improve reasoning paths, and scaling to larger models or diverse datasets will enhance generalization and practical deployment.
AI Executive Summary
Recent advances in large language models (LLMs) have revolutionized natural language understanding, but extending this success to multimodal domains, especially audio-text reasoning, remains challenging. Existing models often struggle with the semantic gap between acoustic signals and textual representations, leading to degraded reasoning performance in audio-conditioned tasks. Traditional training paradigms, relying on external teachers or large annotated datasets, face limitations in scalability and data efficiency. To address these issues, Jing Hu and colleagues propose CORD, a unified, in-model, multi-level alignment framework that performs online self-distillation guided by importance-aware token weighting and global reward optimization.
The core innovation lies in combining token-level importance-weighted reverse KL divergence with a judge-based sequence-level reward, optimized via Group Relative Policy Optimization (GRPO). This design allows the model to dynamically correct semantic deviations during inference, focusing on early and critical tokens that influence reasoning outcomes. The approach leverages the model’s internal text modality as a teacher, eliminating the need for external supervision and reducing off-policy bias.
Extensive experiments on benchmarks like Qwen2-Audio-7B-Instruct and Step-Audio2-mini demonstrate that CORD substantially narrows the audio-text reasoning gap—by over 40%—and nearly achieves parity with text-conditioned performance. The results highlight the method’s data efficiency, requiring only 80K synthetic samples, and its robustness across diverse reasoning tasks. This work marks a significant step toward scalable, high-performance multimodal AI, with promising applications in speech understanding, intelligent QA, and beyond.
Despite these advances, challenges remain in deploying the framework in real-world noisy environments and complex reasoning scenarios. Future work will focus on integrating real audio data, refining global reward models, and expanding to multi-task, multi-modal systems, aiming to realize truly versatile and resilient multimodal AI solutions.
Deep Analysis
Background
Multimodal learning has rapidly evolved, with models like CLIP and VATT demonstrating effective cross-modal alignment via large-scale pretraining. However, in audio-text reasoning, performance disparities persist due to modality-specific feature gaps and limited data. Traditional approaches, such as representation-level alignment and synthetic data augmentation, have achieved limited success, often constrained by data scarcity and reliance on external teachers. Recent efforts explore end-to-end training and self-distillation, but challenges remain in dynamically correcting semantic misalignments during inference, especially in reasoning-intensive tasks. Addressing these issues is crucial for advancing speech understanding, human-computer interaction, and multimodal reasoning systems.
Core Problem
The main challenge is the persistent semantic misalignment between audio and text modalities, leading to degraded reasoning accuracy in audio-conditioned tasks. Existing methods rely on static alignment or external teachers, which are insufficient for dynamic correction during inference. This results in a performance gap that widens under data constraints. The difficulty lies in designing a unified framework that can adaptively and efficiently align reasoning trajectories at multiple granularities within a single model, ensuring semantic fidelity and robustness across diverse tasks. Overcoming this bottleneck is essential for deploying reliable multimodal AI in real-world scenarios.
Innovation
The paper introduces several key innovations:
1) Importance-aware token-level reverse KL divergence, which prioritizes correcting high-discrepancy tokens, especially early in reasoning, to prevent error propagation.
2) Judge-guided sequence-level reward, enabling global trajectory alignment by evaluating semantic consistency between audio and text outputs.
3) End-to-end, in-model self-distillation that combines local and global alignment strategies within a single framework, avoiding external teacher models and off-policy issues.
These innovations collectively enable the model to dynamically self-correct during inference, significantly reducing the modality gap and improving reasoning accuracy.
Methodology
- ��Input：语音和文本的语义等价对。
- ��模型架构：基于Transformer，结合音频编码器和文本编码器。
- ��令牌层对齐：利用逆KL散度衡量音频与文本在每个解码步骤的偏差，选择前K个高偏差令牌进行加权修正。
- ��加权机制：引入重要性感知的逆KL权重和位置衰减，强调早期关键令牌。
- ��序列层对齐：利用判别模型评估完整推理路径的语义一致性，结合GRPO优化全局推理轨迹。
- ��训练流程：在80K合成样本上，采用端到端动态自蒸馏，实时调整模型参数以缩小模态差距。
Experiments
- ��数据：使用NuminaMath合成的80K音频-文本对，涵盖数学推理任务。
- ��基线：包括SFT、前向KL蒸馏等。
- ��指标：推理准确率、模态差距百分比。
- ��设置：调节超参数α、β，采用AdamW优化，模型在多任务、多场景中测试。
- ��评估：在多模态推理、知识问答和数学任务上验证性能，进行消融分析确认机制有效性。
Results
- ��在Qwen2-Audio-7B-Instruct上，CORD将模态差距从70.2%降低到不足30%，提升显著；在Step-Audio2-mini上，差距缩小至6%，几乎实现模态一致。•在多个任务中，模型不仅提升了准确率，还增强了复杂推理和多模态理解的鲁棒性。•消融实验显示，令牌加权和全局奖励机制的结合是性能提升的关键，验证了设计的合理性。
Applications
- ��即时应用：可用于智能语音助手、语音问答系统，提升理解和交互能力。•长远目标：推动多模态AI在教育、医疗、自动驾驶等行业的广泛应用，实现更自然、更智能的人机交互。
Limitations & Outlook
- ��模型依赖合成数据，实际环境中噪声和多样性影响尚未充分解决。•判别模型可能引入偏差，影响推理的全面性。•在极端复杂推理任务中仍存在性能瓶颈，未来需结合多源信息和更强特征提取技术进行优化。
Plain Language Accessible to non-experts
想象你在厨房做饭，食材代表信息，厨师代表模型。传统方法就像用一份食谱，按步骤操作，但不同厨师理解不同，味道也会不同。现在，厨师自己不断尝试、品尝、调整调料，逐步改进菜肴。CORD就像这个厨师，它在做饭时会自己不断试味、调整，确保每一步都做得更好，最终做出味道一致的菜。它特别关注那些关键的调料和早期步骤，避免偏差累积。这样，菜就能既好吃又一致，模型也是如此，更懂得理解和推理音频信息。
ELI14 Explained like you're 14
想象你在学校听老师讲课，有时候讲得太快，你跟不上，特别是重点内容。传统的方法就像用讲义，自己反复看，但还是会漏掉。CORD就像个聪明的同学，他会自己偷偷复述老师讲的内容，特别关注那些重要的词和句子。如果发现自己理解错了，他会自己调整，确保理解正确。整个学习过程中，他不断检查自己，确保每个部分都理解得差不多了。这样一来，不仅学得更快，还能抓住重点，考试也会考得更好。它用一种聪明的“自我修正”方法，让学习变得更有效率。
Glossary
逆KL散度 (Reverse KL Divergence)
衡量两个概率分布偏差的指标，强调高概率区域的匹配，适用于修正模型偏差。
在令牌层面，用于衡量音频与文本条件下的生成偏差。
GRPO (Group Relative Policy Optimization)
一种基于群组的相对优势策略优化方法，用于全局推理轨迹的优化。
在序列层面，通过判别模型奖励引导推理路径。
自蒸馏 (Self-distillation)
模型利用自身内部信息作为教师，进行多层次、多粒度的知识迁移。
实现跨模态对齐，减少对外部教师的依赖。
模态差距 (Modality Gap)
不同模态（如音频与文本）在语义表达上的偏差，影响模型推理性能。
本文旨在缩小音频与文本模态的推理差距。
判别模型 (Discriminator Model)
用于评估两个序列语义一致性的模型，提供全局奖励信号。
引导推理路径的全局优化。
Open Questions Unanswered questions from this research
- 1 如何在真实环境中应对音频噪声和多样性，提升模型鲁棒性仍是挑战。
- 2 判别模型的偏差可能影响整体推理质量，未来需设计更公正的评估机制。
- 3 在极端复杂推理任务中，模型表现仍有限，需结合多源特征进行优化。
Applications
Immediate Applications
智能语音助手
提升语音助手在复杂问答和推理中的表现，使其更智能、更自然。
Long-term Vision
多模态人机交互
实现更自然、更智能的多模态交互系统，推动教育、医疗等行业变革。
Abstract
Large Audio Language Models (LALMs) have garnered significant research interest. Despite being built upon text-based large language models (LLMs), LALMs frequently exhibit a degradation in knowledge and reasoning capabilities. We hypothesize that this limitation stems from the failure of current training paradigms to effectively bridge the acoustic-semantic gap within the feature representation space. To address this challenge, we propose CORD, a unified alignment framework that performs online cross-modal self-distillation. Specifically, it aligns audio-conditioned reasoning with its text-conditioned counterpart within a unified model. Leveraging the text modality as an internal teacher, CORD performs multi-granularity alignment throughout the audio rollout process. At the token level, it employs on-policy reverse KL divergence with importance-aware weighting to prioritize early and semantically critical tokens. At the sequence level, CORD introduces a judge-based global reward to optimize complete reasoning trajectories via Group Relative Policy Optimization (GRPO). Empirical results across multiple benchmarks demonstrate that CORD consistently enhances audio-conditioned reasoning and substantially bridges the audio-text performance gap with only 80k synthetic training samples, validating the efficacy and data efficiency of our on-policy, multi-level cross-modal alignment approach.