PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition

The evolution of audio recognition has transitioned from traditional handcrafted features like MFCCs and energy measures to deep learning-based approaches, which have demonstrated superior performance. Early models relied on GMMs, HMMs, and SVMs, but these struggled with complex, real-world data. The advent of CNNs enabled automatic feature learning from spectrograms, significantly improving accuracy. Large datasets such as AudioSet, released by Google, provided the scale needed for training deep models. Despite progress, existing models often lack generalization across diverse tasks and are limited by small datasets. Transfer learning has shown promise, but systematic large-scale pretraining for audio remains underexplored. This work addresses this gap by training multiple CNN architectures on AudioSet, aiming to develop versatile, high-capacity models capable of transfer to various audio recognition tasks.

The core challenge lies in effectively leveraging massive, diverse audio datasets to build models that generalize well across multiple tasks. Existing models tend to overfit small datasets, struggle with noisy environments, and lack robustness. Additionally, capturing both spectral and temporal features efficiently remains difficult. The need for scalable, adaptable models that can perform well in real-world scenarios and transfer knowledge across tasks is pressing. Overcoming these bottlenecks requires innovative architectures, effective training strategies, and multi-modal feature integration, which this work aims to provide.

First, the paper introduces a comprehensive set of CNN architectures pretrained on AudioSet, including VGG, ResNet, and MobileNet, establishing a large-scale training baseline. Second, it proposes Wavegram-Logmel-CNN, a novel fusion of learned time-frequency features and traditional spectrograms, enhancing frequency pattern recognition. Third, it employs advanced data augmentation techniques like Mixup and SpecAugment to improve robustness. Fourth, the study demonstrates effective transfer learning strategies—feature extraction and fine-tuning—across multiple audio tasks, validating the models' versatility. These innovations collectively push the boundary of scalable, multi-task audio recognition.

• �� Build multiple CNN architectures (VGG, ResNet, MobileNet) tailored for audio data, with modifications for large-scale training. • Pretrain models on AudioSet, employing data balancing to address class imbalance. • Extract features using Wavegram via 1D convolutional layers on raw waveforms, capturing frequency information. • Combine Wavegram and Logmel spectrograms along the channel dimension to form a fused input, leveraging both learned and handcrafted features. • Use data augmentation techniques such as Mixup (interpolating between samples) and SpecAugment (masking in frequency and time domains) to improve generalization. • Train models with a two-stage process: initial pretraining followed by transfer learning (feature extraction or fine-tuning) on target tasks. • Evaluate performance using metrics like mAP, AUC, across multiple datasets and tasks to validate transferability.

The models are trained on the AudioSet dataset, containing over 2 million clips, with 80% for training and 20% for validation. Hyperparameters include a learning rate schedule with Adam optimizer, batch size of several hundred, and early stopping. Ablation studies compare architectures, feature fusion strategies, and augmentation effects. Transfer experiments involve applying pretrained models to acoustic scene classification, music tagging, and speech emotion recognition datasets, with fine-tuning or feature extraction. Performance metrics such as mAP, AUC, and F1-score are used. Results show that Wavegram-Logmel-CNN outperforms baseline models, with significant improvements in robustness and accuracy, validating the effectiveness of large-scale pretraining and feature fusion.

The pretrained models achieve a peak mAP of 0.439 on AudioSet, surpassing previous bests (~0.392). Transfer to other tasks yields state-of-the-art results, with improvements of 10-15% over existing systems. ResNet54 offers the best accuracy but at higher computational cost, while MobileNetV2 provides a good trade-off. Ablation results confirm that Wavegram enhances frequency pattern learning, and data augmentation techniques significantly improve robustness, especially in noisy conditions. Cross-task evaluations demonstrate the models' strong generalization, confirming the value of large-scale pretraining.

The pretrained models can be directly applied to real-world scenarios such as surveillance, content moderation, voice assistants, and multimedia indexing. They require minimal task-specific data for fine-tuning, reducing development time and cost. The models' robustness makes them suitable for deployment in diverse environments, including mobile and edge devices. Long-term, these models could underpin multi-modal AI systems, enabling more natural human-computer interactions and automated content analysis across industries.

Despite high accuracy, models still face challenges in noisy, real-world environments with overlapping sounds. Large-scale training demands substantial computational resources, limiting accessibility. The reliance on labeled datasets like AudioSet introduces potential biases and label noise, affecting transfer performance. Future work should focus on improving robustness, reducing computational costs, and exploring semi-supervised learning to mitigate data annotation bottlenecks.