Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
Qwen-Audio scales multi-task training with hierarchical tags, supporting 30+ tasks and surpassing SOTA models.
Key Findings
Methodology
Qwen-Audio integrates a single audio encoder with the Qwen-7B language model. It employs hierarchical tags to mitigate interference across datasets, enabling multi-task training across 30+ tasks, including ASR, audio QA, and music analysis.
Key Results
- Achieved SOTA performance on Aishell1 (WER 1.69%), ClothoAQA, and VocalSound benchmarks without task-specific fine-tuning.
- Improved CoVoST2 multilingual translation BLEU score to 35.2, outperforming Pengi by 2.1 points.
- SRWT task enhanced timestamp alignment, significantly boosting ASR and audio QA performance.
Significance
This work addresses the limitations of existing audio-language models by enabling diverse task and audio type support. Its open-source nature fosters advancements in multimodal AI research and applications.
Technical Contribution
Introduced a multi-task training framework with hierarchical tags for knowledge sharing and interference reduction. Pioneered the inclusion of SRWT tasks in multi-task audio models, enhancing timestamp alignment and task performance.
Novelty
Qwen-Audio is the first model to unify diverse audio types and tasks within a single framework, addressing dataset interference and introducing SRWT for improved temporal alignment.
Limitations
- Performance on low-resource languages remains suboptimal.
- Struggles with extremely long audio or complex noisy environments.
- High computational cost for multi-task training.
Future Work
Future work includes optimizing training efficiency, improving low-resource language support, and handling more complex audio scenarios.
AI Executive Summary
Qwen-Audio is a universal audio-language model that overcomes the limitations of existing models by supporting diverse audio types and tasks through a novel multi-task training framework. Its key innovation lies in hierarchical tags, which prevent dataset interference, and the SRWT task, which enhances temporal alignment.
Experimental results demonstrate that Qwen-Audio achieves SOTA performance on benchmarks like Aishell1 and ClothoAQA without task-specific fine-tuning. Its multilingual and multi-task capabilities excel in ASR, audio QA, and music analysis, outperforming models like Pengi.
This research advances multimodal AI by providing a robust open-source tool for academia and industry. However, challenges remain in low-resource languages and complex audio scenarios, highlighting opportunities for future optimization and expansion.
Deep Analysis
Background
Recent advancements in large language models have revolutionized text processing but remain limited in multimodal understanding. Audio, encompassing speech, natural sounds, and music, presents unique challenges due to its diverse nature. Existing models often focus on single tasks like ASR or translation.
Core Problem
Current audio-language models lack the ability to handle diverse audio types and tasks simultaneously. Multi-task training often suffers from dataset interference, limiting generalization and practical applicability.
Innovation
Qwen-Audio introduces hierarchical tags to resolve dataset interference and employs a single audio encoder to support diverse audio types. The inclusion of SRWT tasks enhances temporal alignment, a novel addition to multi-task audio models.
Methodology
- �� Initialized audio encoder with Whisper-large-v2 for diverse audio processing.
- �� Integrated Qwen-7B for multilingual and multi-task capabilities.
- �� Designed hierarchical tags (task, language, timestamp) to prevent interference.
- �� Pretrained on 30+ datasets covering ASR, audio QA, and music analysis.
Experiments
Evaluated on datasets like Aishell1, ClothoAQA, and CoVoST2. Baselines included Pengi and SpeechT5. Metrics included WER for ASR and BLEU for translation.
Results
Qwen-Audio achieved WER of 1.69% on Aishell1 and BLEU of 35.2 on CoVoST2, outperforming Pengi. SRWT significantly improved timestamp alignment, enhancing ASR and QA tasks.
Applications
Applicable to voice assistants, audio content generation, and music analysis. Suitable for multilingual, multi-task environments.
Limitations & Outlook
Limited performance on low-resource languages and complex audio scenarios. High training costs necessitate efficiency improvements.
Plain Language Accessible to non-experts
Think of Qwen-Audio as a 'universal translator' for audio. It can understand speech, music, and natural sounds, like recognizing a bird's chirp or translating a Chinese sentence into English. Using 'hierarchical tags,' it knows exactly what task to focus on, like a multitasking expert who switches seamlessly between roles.
ELI14 Explained like you're 14
Imagine you have a super-smart friend who can listen to any language, tell you what instruments are in a song, or even describe the mood of a bird's chirping! That's Qwen-Audio. It can translate Chinese to English or explain what's happening in a sound clip. But it still needs more training to handle rare languages or noisy sounds.
Glossary
Multi-task Learning
A method where a model is trained to perform multiple tasks simultaneously, improving knowledge sharing.
Used to train Qwen-Audio across 30+ tasks.
Hierarchical Tags
A tagging system to distinguish tasks, languages, and timestamps, reducing dataset interference.
Key to Qwen-Audio's multi-task framework.
SRWT (Speech Recognition with Word-level Timestamps)
An ASR task requiring word-level timestamp predictions.
Improved temporal alignment in Qwen-Audio.
Audio Encoder
A module that processes audio signals into features usable by the model.
Qwen-Audio uses Whisper-large-v2 as its encoder.
BLEU Score
A metric for evaluating translation quality; higher scores indicate better performance.
Used to assess CoVoST2 translation tasks.
Open Questions Unanswered questions from this research
- 1 How can performance on low-resource languages be improved?
- 2 Can multi-task training costs be reduced further?
- 3 How to enhance robustness in noisy environments?
Applications
Immediate Applications
Voice Assistants
Supports multilingual ASR and translation for enhanced user experiences.
Audio Content Generation
Generates high-quality audio descriptions or subtitles for podcasts or videos.
Long-term Vision
Multimodal Interaction
Integrates audio, text, and images for next-gen human-computer interaction.
Abstract
Recently, instruction-following audio-language models have received broad attention for audio interaction with humans. However, the absence of pre-trained audio models capable of handling diverse audio types and tasks has hindered progress in this field. Consequently, most existing works have only been able to support a limited range of interaction capabilities. In this paper, we develop the Qwen-Audio model and address this limitation by scaling up audio-language pre-training to cover over 30 tasks and various audio types, such as human speech, natural sounds, music, and songs, to facilitate universal audio understanding abilities. However, directly co-training all tasks and datasets can lead to interference issues, as the textual labels associated with different datasets exhibit considerable variations due to differences in task focus, language, granularity of annotation, and text structure. To overcome the one-to-many interference, we carefully design a multi-task training framework by conditioning on a sequence of hierarchical tags to the decoder for encouraging knowledge sharing and avoiding interference through shared and specified tags respectively. Remarkably, Qwen-Audio achieves impressive performance across diverse benchmark tasks without requiring any task-specific fine-tuning, surpassing its counterparts. Building upon the capabilities of Qwen-Audio, we further develop Qwen-Audio-Chat, which allows for input from various audios and text inputs, enabling multi-turn dialogues and supporting various audio-central scenarios.