AudioLDM: Text-to-Audio Generation with Latent Diffusion Models
AudioLDM employs latent diffusion models trained on audio-only data with CLAP embeddings, achieving state-of-the-art text-to-audio generation on a single GPU.
Key Findings
Methodology
AudioLDM integrates a variational autoencoder (VAE) to encode audio into a continuous latent space, combined with a contrastive language-audio pretraining (CLAP) to embed both text and audio into a shared cross-modal space. A latent diffusion model (LDM) is trained on audio embeddings, conditioned on CLAP-derived text embeddings during sampling. The training employs data augmentation (mixup) on audio data, removing the need for paired audio-text datasets. During inference, the model generates audio by reversing the diffusion process conditioned on text embeddings, enabling zero-shot manipulation tasks like style transfer, inpainting, and super-resolution.
Key Results
- On the AudioCaps dataset, AudioLDM achieves a Frechet Distance (FD) of 23.31, outperforming DiffSound (47.68) by a large margin, trained on a single GPU. The model surpasses previous open-source systems in both objective metrics (FD, IS, KL) and subjective evaluations, demonstrating high-quality, controllable, and versatile audio synthesis.
- By leveraging CLAP embeddings, the system avoids reliance on large-scale paired datasets, training solely on audio data with data augmentation, which enhances robustness and generalization. Larger models and expanded datasets further improve performance, confirming the scalability of the approach.
- The system supports multiple zero-shot audio manipulations, including style transfer, super-resolution, and inpainting, with high fidelity and relevance to input text, validated through both quantitative metrics and human assessments.
Significance
This work advances the field of text-to-audio synthesis by introducing a highly efficient, high-quality, and versatile framework that circumvents the data scarcity problem. The integration of CLAP embeddings with latent diffusion models enables effective cross-modal learning and zero-shot control, opening new avenues for personalized audio content creation. Its ability to perform multiple audio manipulations without fine-tuning significantly broadens practical applications in entertainment, virtual reality, and content production. The approach sets a new benchmark for efficiency and quality, making real-time, controllable audio generation feasible on modest hardware, thus democratizing access to advanced audio synthesis technology.
Technical Contribution
The core technical innovation lies in applying continuous latent diffusion models within a shared cross-modal embedding space, trained solely on audio data augmented via mixup, eliminating the dependency on paired datasets. The use of CLAP embeddings as conditions during sampling decouples language understanding from generative modeling, enabling zero-shot tasks. The model architecture combines a VAE for compression, a CLAP-based cross-modal embedding space, and a latent diffusion process optimized with a noise estimation loss. The integration of classifier-free guidance further enhances controllability and diversity of generated outputs. These contributions collectively push the boundaries of efficiency, controllability, and quality in audio synthesis.
Novelty
This is the first work to successfully implement a continuous latent diffusion model conditioned solely on audio embeddings and CLAP-based text embeddings for high-fidelity, zero-shot text-to-audio generation. Unlike prior methods relying on discrete representations or requiring extensive paired datasets, this approach leverages the shared embedding space for training and inference, enabling high-quality generation with minimal data. Its multi-task capabilities, including style transfer and inpainting, without fine-tuning, represent a significant innovation in the field of multimodal generative modeling.
Limitations
- The model's performance diminishes with highly complex or long-duration audio scenes, where the latent space may not fully capture intricate details, leading to artifacts or loss of fidelity.
- Training large models still requires significant computational resources, which may limit accessibility for some researchers or practitioners.
- While zero-shot manipulation is supported, fine-grained control over specific attributes still needs further development to meet professional standards.
Future Work
Future directions include enhancing the model's capacity to handle complex, multi-source audio scenes, improving fine-grained controllability, and reducing computational costs further. Integrating multimodal cues such as visual context could expand application scenarios. Additionally, exploring unsupervised or semi-supervised training strategies may further reduce data requirements, making the technology more accessible and scalable for real-world deployment.
AI Executive Summary
The rapid evolution of audio synthesis technologies has opened new horizons for personalized content creation, virtual reality, and entertainment. However, existing approaches often rely on extensive paired datasets and high computational costs, limiting their practicality and scalability. Haohe Liu and colleagues address these challenges with AudioLDM, a novel framework that leverages latent diffusion models conditioned on CLAP embeddings to generate high-quality, controllable audio from text prompts. This system is trained solely on audio data, employing data augmentation techniques like mixup to enhance robustness, and does not require large-scale paired datasets. The core innovation is the use of a variational autoencoder (VAE) to encode audio into a continuous latent space, combined with a latent diffusion process that operates efficiently within this space. During inference, the model can generate diverse audio outputs conditioned on text embeddings, supporting zero-shot tasks such as style transfer, super-resolution, and inpainting.
Experimental results demonstrate that AudioLDM achieves a Frechet Distance of 23.31 on the AudioCaps dataset, outperforming previous methods like DiffSound by a wide margin. The model's ability to produce high-fidelity, relevant audio with minimal computational resources marks a significant step forward in the field. Its versatility in multi-task zero-shot manipulation opens new avenues for creative applications, from personalized sound effects to adaptive virtual environments.
This work not only advances the technical state-of-the-art but also democratizes high-quality audio synthesis by reducing hardware requirements and data dependencies. The integration of cross-modal embeddings and latent diffusion models paves the way for more intuitive, flexible, and scalable audio generation systems. Despite current limitations in handling highly complex scenes, ongoing research aims to improve controllability, reduce costs, and extend capabilities to more diverse scenarios, promising a future where personalized, real-time audio content becomes ubiquitous.
Deep Dive
Abstract
Text-to-audio (TTA) system has recently gained attention for its ability to synthesize general audio based on text descriptions. However, previous studies in TTA have limited generation quality with high computational costs. In this study, we propose AudioLDM, a TTA system that is built on a latent space to learn the continuous audio representations from contrastive language-audio pretraining (CLAP) latents. The pretrained CLAP models enable us to train LDMs with audio embedding while providing text embedding as a condition during sampling. By learning the latent representations of audio signals and their compositions without modeling the cross-modal relationship, AudioLDM is advantageous in both generation quality and computational efficiency. Trained on AudioCaps with a single GPU, AudioLDM achieves state-of-the-art TTA performance measured by both objective and subjective metrics (e.g., frechet distance). Moreover, AudioLDM is the first TTA system that enables various text-guided audio manipulations (e.g., style transfer) in a zero-shot fashion. Our implementation and demos are available at https://audioldm.github.io.