Unified Audio Intelligence Without Regressing on Text Intelligence
Audex, built on Nemotron-Cascade-2-30B-A3B, unifies audio-text modeling with a single Transformer decoder, achieving state-of-the-art performance in audio understanding, speech recognition, translation, and generation, while maintaining text reasoning.
Key Findings
Methodology
Audex employs a unified Transformer decoder architecture, encoding audio inputs via AF-Whisper into continuous embeddings, then projecting into the text embedding space. It extends the vocabulary with discrete audio tokens and predicts text and audio tokens autoregressively. The training involves multi-stage supervised learning on 157.4B audio tokens and 320.5B text tokens, combined with text-only RL and multi-domain distillation to preserve reasoning and knowledge capabilities. This design enables seamless multimodal fusion and generation across speech, music, and environmental sounds, outperforming existing models in multiple benchmarks.
Key Results
- Audex achieves near state-of-the-art results in audio understanding, with performance comparable to Step-Audio-R1.1-33B and Qwen-Omni-30B-A3B-Thinking, and surpasses them in speech recognition (WER 7.91%). It supports long-context reasoning up to 1 million tokens, maintaining reasoning and knowledge capabilities comparable to its text-only backbone. In multimodal generation, it produces high-quality speech, music, and environmental sounds, demonstrating versatility across tasks.
- In reasoning benchmarks like IMO AnswerBench and HMMT, Audex matches or exceeds the performance of its text-only backbone, showing that multimodal fusion does not impair reasoning. It also excels in translation, TTS, and speech-to-speech tasks, confirming its broad applicability.
- The architecture’s core innovation lies in a single decoder handling both modalities, with extended vocabulary and discrete tokens, combined with multi-stage training, resulting in a model that balances multimodal capabilities with strong reasoning and knowledge retention.
Significance
This work addresses a critical challenge in AI: building models that can understand and generate multiple modalities without sacrificing reasoning and knowledge abilities. By maintaining the core strengths of text-based LLMs while integrating audio modalities, Audex paves the way for more general-purpose AI systems. Its simple yet effective architecture offers scalability and compatibility with existing training infrastructure, making it a significant step toward truly multimodal artificial general intelligence with broad industrial and academic implications.
Technical Contribution
The key technical advance is a single Transformer decoder that directly predicts both text and discretized audio tokens, enabled by vocabulary expansion and audio encoding. The multi-stage training process, combining supervised learning, RL, and distillation, ensures the model retains reasoning, knowledge, and long-context understanding. This approach circumvents the performance degradation seen in previous multimodal models, setting a new standard for integrated multimodal large models.
Novelty
This is the first work to unify audio and text processing within a single Transformer decoder architecture, leveraging an extended vocabulary and discrete audio tokens. It effectively preserves reasoning and knowledge capabilities while enabling high-quality multimodal generation, addressing the common trade-off between multimodal fusion and reasoning performance seen in prior models.
Limitations
- Despite impressive results, the model’s robustness in noisy or reverberant environments remains limited, mainly due to the audio encoder’s constraints. Computational costs are high, requiring extensive data and hardware resources. The quality of non-speech audio generation still needs improvement for high-fidelity applications. Future work should focus on enhancing robustness, efficiency, and output quality, especially for complex real-world scenarios.
Future Work
Next steps include optimizing model efficiency, improving robustness in challenging acoustic environments, and expanding multimodal outputs to include visual data. Further research will explore adaptive training strategies, smaller model variants, and real-time inference capabilities, aiming to develop more versatile, scalable, and robust multimodal AI systems.
AI Executive Summary
The rapid evolution of artificial intelligence has highlighted the importance of multimodal understanding, especially integrating audio and text. Existing models often face a trade-off: enhancing multimodal capabilities tends to degrade core reasoning and knowledge abilities inherited from large language models (LLMs). Addressing this challenge, the paper introduces Audex, a unified audio-text large model built upon Nemotron-Cascade-2-30B-A3B. Unlike traditional approaches that stack separate modules, Audex employs a single Transformer decoder architecture, which encodes audio inputs into a shared embedding space and predicts both text and discretized audio tokens autoregressively.
This design allows seamless fusion of modalities, enabling the model to perform a wide range of tasks including audio understanding, speech recognition, translation, text-to-speech (TTS), and audio generation, with performance surpassing or matching state-of-the-art models. The training process involves meticulous data curation, combining 157.4 billion audio tokens and 320.5 billion text tokens, and multi-stage supervised learning, followed by reinforcement learning and multi-domain distillation. These strategies ensure that the model retains the reasoning, knowledge, and long-context capabilities of its text-only backbone.
Experimental results demonstrate that Audex achieves near state-of-the-art results in multiple benchmarks, with a speech recognition error rate of 7.91%, and supports long-context reasoning up to 1 million tokens. Its ability to generate high-quality speech, music, and environmental sounds across diverse tasks showcases its versatility. The architecture’s simplicity, efficiency, and compatibility with existing training infrastructure mark a significant advancement in multimodal AI.
Looking ahead, future research will focus on improving robustness in noisy environments, reducing computational costs, and expanding multimodal outputs to include visual data. Overall, Audex represents a crucial step toward more integrated, capable, and scalable artificial intelligence systems, bridging the gap between language and perception in a unified framework.
Deep Dive
Glossary
Transformer (变换器)
一种深度学习模型架构，擅长处理序列数据，广泛应用于自然语言处理和多模态任务。
本文中用于统一处理音频和文本的解码器架构。
Mixture-of-Experts (专家混合模型)
一种模型结构，通过多个子模型（专家）协作，提高模型容量和效率。
Nemotron-Cascade-2采用的核心架构之一。
离散音频token
将连续音频信号量化为离散符号，用于模型预测和生成。
模型通过扩展词汇表预测音频tokens，实现多模态生成。
多阶段监督学习
分阶段训练策略，逐步加入不同任务，优化模型性能。
训练过程中用于融合多模态信息，保持推理能力。
文本强化学习 (RL)
利用奖励信号优化模型生成的文本质量和一致性。
用于提升模型的推理和知识保持能力。
Open Questions Unanswered questions from this research
- 1 如何进一步提升模型在极端噪声环境下的鲁棒性仍是未解难题，尤其在复杂环境中音频编码器的表现有限。未来需探索更高效的训练策略和模型结构，以降低成本并增强多模态融合的适应性。
Applications
Immediate Applications
智能语音助手
支持多模态交互，结合语音识别、翻译和内容生成，提升用户体验。
多模态内容创作
自动生成配音、音乐和环境声，为内容制作提供智能工具。
Long-term Vision
通用多模态AI系统
融合视觉、听觉和语言，实现更全面的智能感知与交互，推动人机交互革命。
Abstract
Audio intelligence involves understanding, reasoning about, and generating both audio and speech. In this work, we introduce Nemotron-Labs-Audex-30B-A3B (Audex), a unified audio-text LLM built on Nemotron-Cascade-2-30B-A3B, a strong text-only MoE LLM. Audex adopts a simple unified design with a single Transformer decoder: audio inputs are encoded and projected into the text embedding space, while text tokens and quantized audio output tokens are treated uniformly during generation. This architecture enables strong audio-text fusion, seamless multimodal generation, and compatibility with standard LLM training and inference infrastructure. For training, we meticulously curate audio-text datasets comprising 157.4B audio tokens and 320.5B text tokens. We apply multi-stage supervised training on these datasets, followed by text-only Cascade RL and multi-domain on-policy distillation. Audex delivers state-of-the-art audio understanding, speech recognition and translation, text-to-speech, audio generation, and speech-to-speech generation, while preserving very compelling reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only LLM backbone with marginal or no regression. We release the model checkpoints to facilitate open research.