UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions
UniSonate model generates speech, music, and sound effects via text instructions with a TTS WER of 1.47%.
Key Findings
Methodology
UniSonate uses a conditional flow matching framework with dynamic token injection to project unstructured sound effects into a structured temporal latent space, coupled with a multi-stage curriculum learning strategy to mitigate cross-modal optimization conflicts.
Key Results
- UniSonate achieved a WER of 1.47% in instruction-based TTS, significantly outperforming single-task baselines.
- In TTM, it scored 3.18 in SongEval coherence, showing superior structural consistency.
- Maintained competitive fidelity in TTA.
Significance
UniSonate unifies the generation of speech, music, and sound effects, addressing long-standing challenges in cross-modal integration, significantly enhancing the structural coherence and prosodic expressiveness of generated audio.
Technical Contribution
The model is the first to achieve unified generation of multiple audio modalities via text instructions without reference audio, introducing dynamic token injection and a multi-stage curriculum learning strategy.
Novelty
UniSonate is the first model to generate speech, music, and sound effects through a unified interface without reference audio, overcoming the integration challenges of structured and unstructured semantic representations.
Limitations
- The model may underperform in extremely complex sound effect scenarios due to its reliance on token length inference.
- Generated audio may lack precision in some details.
Future Work
Future work could explore more complex sound effect scenarios, optimize the dynamic token injection mechanism, and expand multilingual support.
AI Executive Summary
The field of audio generation has long been fragmented into specialized tasks like Text-to-Speech (TTS), Text-to-Music (TTM), and Text-to-Audio (TTA), each with its unique control paradigms. UniSonate addresses these inconsistencies by introducing a unified flow matching framework. The model employs a dynamic token injection mechanism to project unstructured environmental sounds into a structured temporal latent space, achieving precise duration control. Coupled with a multi-stage curriculum learning strategy, it effectively mitigates cross-modal optimization conflicts. Experimental results show that UniSonate achieves a WER of 1.47% in instruction-based TTS, a SongEval coherence score of 3.18 in TTM, and maintains competitive fidelity in TTA. This research holds significant academic and industrial implications, particularly for generating complex audio scenes like dialogues with background music and environmental effects. However, the model may underperform in extremely complex sound effect scenarios. Future research could further optimize the dynamic token injection mechanism and expand multilingual support.
Deep Analysis
Background
Recent advances in audio generation have significantly improved Text-to-Speech (TTS), Text-to-Music (TTM), and Text-to-Audio (TTA) technologies. However, these tasks are often treated as separate research directions, each with its control paradigm, making it challenging to achieve unified cross-modal generation. Existing attempts like Vevo2 and CosyVoice have made progress in unifying speech and singing but still rely on reference audio, lacking the flexibility of natural language descriptions.
Core Problem
The core problem is how to generate speech, music, and sound effects within a unified framework. Speech and music require precise temporal alignment, while sound effects are inherently holistic and unstructured, lacking rigid temporal boundaries. Training a model on concatenated datasets often leads to negative transfer, where the variance of unstructured sound effects destabilizes the articulation required for high-quality speech.
Innovation
UniSonate's core innovation lies in introducing a dynamic token injection mechanism that projects unstructured environmental sounds into a structured temporal latent space, achieving precise duration control. The model also employs a multi-stage curriculum learning strategy, gradually expanding from structured speech to semi-structured music and finally to unstructured effects, mitigating optimization conflicts.
Methodology
- �� Utilizes a conditional flow matching framework for unified generation of speech, music, and sound effects. • Dynamic token injection mechanism projects unstructured sound effects into a structured temporal latent space. • Multi-stage curriculum learning strategy expands from speech to music to effects. • Employs a dual-stream Multimodal Diffusion Transformer (MM-DiT) for generation.
Experiments
The experimental design includes training on a large-scale unified audio corpus comprising 50K hours of speech, 20K hours of music, and 1.5M sound effect clips. Evaluations are conducted using standard objective metrics and subjective MOS, with baselines including dedicated TTS, TTM, and TTA models.
Results
UniSonate achieved a WER of 1.47% in instruction-based TTS, a SongEval coherence score of 3.18 in TTM, and maintained competitive fidelity in TTA. Joint training significantly enhanced the structural coherence and prosodic expressiveness of generated speech.
Applications
UniSonate can be applied in scenarios requiring complex audio scene generation, such as dialogues with background music and environmental effects. Its unified interface simplifies the process of multimodal audio generation, making it suitable for various industrial applications.
Limitations & Outlook
The model may underperform in extremely complex sound effect scenarios due to its reliance on token length inference. Additionally, generated audio may lack precision in some details. Future research could further optimize the dynamic token injection mechanism and expand multilingual support.
Plain Language Accessible to non-experts
Imagine you're in a kitchen, and UniSonate is like a master chef who can prepare different dishes simultaneously based on your instructions. You can ask the chef to make a Chinese stir-fry, an Italian pasta, and a fruit juice. Each dish has different preparation and cooking times, but this chef can precisely control the timing and sequence of each dish based on your instructions. Dynamic token injection is like the chef's timer, ensuring each dish is completed at the right time, while the multi-stage curriculum learning strategy is like the chef's training process, gradually improving skills from simple dishes to complex banquets.
ELI14 Explained like you're 14
Imagine you're playing a game, and UniSonate is like a super NPC that can generate different scenes and sound effects based on your instructions. You can have it play a piece of music, create a dialogue scene, or even simulate the sound of a forest. It's like an all-in-one DJ that can precisely control the duration and sequence of each sound effect based on your instructions. Dynamic token injection is like the DJ's beat counter, ensuring each sound effect plays at the right time, while the multi-stage curriculum learning strategy is like the DJ's training process, gradually improving skills from simple beats to complex mixes.
Glossary
UniSonate
A unified audio generation model capable of generating speech, music, and sound effects via text instructions.
UniSonate is the core subject of this paper.
Dynamic Token Injection
A mechanism that projects unstructured sound effects into a structured temporal latent space.
Used for precise duration control of sound effects.
Multi-Stage Curriculum Learning
A learning strategy that gradually expands from structured speech to unstructured effects.
Used to mitigate cross-modal optimization conflicts.
Conditional Flow Matching
A framework for audio generation supporting multimodal generation.
The foundational framework of UniSonate.
Multimodal Diffusion Transformer
An architecture for processing multiple audio modalities.
Used for unified audio generation.
Open Questions Unanswered questions from this research
- 1 How to maintain high-quality generation in extremely complex sound effect scenarios? The current dynamic token injection mechanism may not suffice for these scenarios.
Applications
Immediate Applications
Multimodal Audio Generation
UniSonate can be used to generate complex audio scenes, such as film scores and game sound effects.
Long-term Vision
Intelligent Audio Assistant
In the future, UniSonate could evolve into an intelligent audio assistant, automatically generating and adjusting audio content.
Abstract
Generative audio modeling has largely been fragmented into specialized tasks, text-to-speech (TTS), text-to-music (TTM), and text-to-audio (TTA), each operating under heterogeneous control paradigms. Unifying these modalities remains a fundamental challenge due to the intrinsic dissonance between structured semantic representations (speech/music) and unstructured acoustic textures (sound effects). In this paper, we introduce UniSonate, a unified flow-matching framework capable of synthesizing speech, music, and sound effects through a standardized, reference-free natural language instruction interface. To reconcile structural disparities, we propose a novel dynamic token injection mechanism that projects unstructured environmental sounds into a structured temporal latent space, enabling precise duration control within a phoneme-driven Multimodal Diffusion Transformer (MM-DiT). Coupled with a multi-stage curriculum learning strategy, this approach effectively mitigates cross-modal optimization conflicts. Extensive experiments demonstrate that UniSonate achieves state-of-the-art performance in instruction-based TTS (WER 1.47%) and TTM (SongEval Coherence 3.18), while maintaining competitive fidelity in TTA. Crucially, we observe positive transfer, where joint training on diverse audio data significantly enhances structural coherence and prosodic expressiveness compared to single-task baselines. Audio samples are available at https://qiangchunyu.github.io/UniSonate/.