Diffsound: Discrete Diffusion Model for Text-to-sound Generation

The evolution of sound synthesis has transitioned from rule-based and signal processing methods to deep learning approaches. Early models like spectral-based synthesis and concatenative methods lacked flexibility. The advent of GANs, such as WaveGAN and SpecGAN, improved realism but faced stability issues. Recent progress with VQ-VAE and autoregressive models like Transformer-based architectures enabled more accurate spectrogram prediction. However, these models suffer from slow inference and error accumulation, especially in complex scenes. Diffusion models, originally successful in image synthesis, have recently been adapted for speech and audio generation, showing promise in improving fidelity and diversity. Nonetheless, their application to discrete sequence generation, like mel-spectrogram tokens, remains underexplored. This paper builds on these developments, aiming to combine the strengths of diffusion models with discrete token prediction for efficient, high-quality sound synthesis.

Existing text-to-sound models predominantly rely on autoregressive decoders, which predict spectrogram tokens sequentially. This approach leads to slow inference times unsuitable for real-time applications, and the step-by-step prediction causes bias and error accumulation, degrading sound fidelity and relevance. Moreover, the limited availability of large-scale paired text-audio datasets constrains model training and generalization. Addressing these issues requires a model capable of rapid inference, robust error correction, and effective utilization of large unpaired audio datasets. The challenge lies in designing a non-autoregressive, iterative refinement mechanism that maintains high fidelity while significantly reducing inference latency.

The paper introduces Diffsound, a novel non-autoregressive framework that employs a discrete diffusion process for mel-spectrogram token prediction. Key innovations include: 1) Applying a multi-step iterative denoising process inspired by diffusion models, enabling simultaneous token prediction and refinement. 2) Incorporating a masking strategy during training to simulate noise corruption, facilitating robust denoising. 3) Leveraging large-scale pretraining on AudioSet with a mask-based text generation strategy to address data scarcity. 4) Combining multi-objective loss functions—reconstruction, adversarial, and semantic—to enhance fidelity and relevance. These innovations collectively enable faster inference, mitigate bias, and improve sound quality compared to traditional autoregressive methods.

• �� Extract text features using a pretrained text encoder (e.g., CLIP).• Encode mel-spectrograms into discrete tokens via VQ-VAE, training with a combination of reconstruction and adversarial losses.• Design a diffusion-based token decoder that predicts all tokens simultaneously, then refines them iteratively:
• During training, corrupt tokens with a masking strategy to simulate noise.
• The model learns to denoise tokens over multiple steps, guided by the corrupted input.
• Use a transition matrix to define the corruption process, enabling flexible noise addition.
• During inference, initialize tokens randomly and perform multiple denoising steps to recover the original sequence.
• �� Fine-tune the pretraining on small-scale text-audio pairs, employing curriculum learning from simple to complex sounds.
• �� Decode the refined tokens into mel-spectrograms with the VQ-VAE decoder, then generate waveforms using a trained vocoder like MelGAN.

The model was trained on AudioSet for pretraining and fine-tuned on AudioCaps. Evaluation used subjective MOS scores and objective metrics such as FID and KL divergence. Ablation studies assessed the impact of the number of diffusion steps, masking strategies, and pretraining. Baselines included autoregressive Transformer models and GAN-based approaches. Hyperparameters like diffusion steps, token length, and learning rates were tuned for optimal performance. The experiments demonstrated that Diffsound achieves a MOS of 3.56, outperforming AR models, with inference speed increased five times. The robustness was validated across diverse sound categories, including environmental and multi-source sounds.

Diffsound significantly improves sound quality and inference speed. MOS scores reached 3.56, compared to 2.786 for AR models. The inference time was reduced by 80%, achieving fivefold speedup. Quantitative metrics like FID and KL confirmed the high fidelity and relevance of generated sounds. Ablation results showed that multi-step refinement and masking strategies are critical for performance. The model generalizes well to unseen sound categories, demonstrating robustness and practical viability for real-time applications.

This framework can be directly used in virtual reality, gaming, and multimedia production to generate immersive environmental sounds from text descriptions. It enables content creators to produce high-fidelity audio without extensive sound design expertise, streamlining workflows. Additionally, it supports real-time interactive systems, such as voice assistants and AR/VR environments, where quick and relevant sound generation enhances user experience. Long-term, this technology could facilitate personalized soundscapes and adaptive audio in smart environments, transforming how we create and experience sound content.

Despite its advantages, Diffsound's performance drops in highly complex multi-source scenes due to limited training data diversity. High-frequency detail restoration remains challenging, sometimes resulting in muffled sounds. The training process is computationally intensive, requiring substantial resources, which limits deployment on edge devices. Future work should focus on model compression, multi-modal integration, and expanding training datasets to improve robustness and efficiency in diverse real-world scenarios.