A Prototypical Triplet Loss for Cover Detection
Introduces multi-pitch features and prototypical triplet loss to improve cover detection accuracy, achieving significant gains on large datasets and live scenarios.
Key Findings
Methodology
This work employs a CNN to embed multi-pitch spectrograms into a feature space, training with a novel Prototypical Triplet Loss that encourages tight clustering of covers of the same work. Input features replace traditional dominant melody with comprehensive multi-pitch representations, extracted via a U-Net architecture. The training leverages a dataset with few covers per work (SHS4-), using online prototype computation and semi-hard negative mining to optimize embedding quality. Evaluation on large-scale retrieval and live song identification demonstrates the method's robustness.
Key Results
- On large dataset lookup, the model with multi-pitch input and prototypical loss achieved a MAP of 0.58, outperforming baseline by 6%, with improved generalization to unseen covers.
- In live song identification, R-precision reached 0.646, surpassing traditional models, confirming the benefit of multi-pitch features in noisy environments.
- The proposed loss function enhances clustering of covers around class prototypes, leading to better recognition of unseen samples and robustness in real-world scenarios.
Significance
This research addresses key challenges in music cover detection under realistic conditions, where data is scarce and variations are high. By integrating richer multi-pitch features and a prototype-based loss, it significantly advances the state-of-the-art, enabling more accurate and scalable music content identification. The approach supports applications in copyright enforcement, content management, and real-time recognition, pushing the industry toward more intelligent and reliable systems.
Technical Contribution
The core innovation lies in replacing single melodic features with multi-pitch spectrograms, capturing bass and harmonic lines. The introduction of a Prototypical Triplet Loss, which models each class by its centroid rather than individual samples, simplifies the clustering task. Combining this with online prototype updates and semi-hard negative mining results in a more discriminative embedding space, outperforming existing triplet-based methods and setting new benchmarks for cover detection.
Novelty
This is the first work to systematically incorporate multi-pitch spectral features into deep cover detection models, coupled with a prototype-based triplet loss. Unlike prior approaches relying solely on dominant melody or handcrafted features, this method enhances robustness and generalization, especially in datasets with limited cover annotations, marking a significant step forward in music similarity learning.
Limitations
- The model's robustness drops in extremely noisy or low-quality recordings, such as live concert environments, indicating a need for better noise-robust feature extraction.
- Scalability remains a concern due to the computational cost of online prototype updates and pairwise distance calculations, especially for real-time large-scale applications.
- Evaluation on datasets with very few covers per work (e.g., only one or two) is limited; further validation is needed for such scenarios, and the model's performance under extreme data scarcity remains uncertain.
Future Work
Future directions include integrating multimodal data like album art and lyrics, developing more efficient online prototype updating algorithms, and expanding datasets to include more diverse genres and languages. Additionally, exploring end-to-end training with raw audio inputs and multi-task learning frameworks could further improve robustness and applicability in real-world systems.
AI Executive Summary
Music cover detection has long been a challenging problem in music information retrieval, complicated by the diversity of cover versions and limited annotated data. Traditional methods relying on handcrafted features or single melodic lines often struggle to generalize across different recording conditions and musical styles. Recent advances in deep learning have improved feature extraction, but the core challenge remains: how to effectively measure similarity between different versions of the same work.
This study introduces a novel approach that leverages multi-pitch spectral representations, capturing bass lines, harmonic content, and melodic contours simultaneously. Using a U-Net architecture, the model extracts rich features from these spectra, which are then embedded into a low-dimensional space via a CNN. To enhance clustering of covers, the authors propose a Prototypical Triplet Loss, where each class (musical work) is represented by a centroid in the embedding space, rather than individual samples. This formulation simplifies the clustering task and improves the model’s ability to generalize to unseen covers.
Extensive experiments on a realistic dataset (SHS4-) demonstrate the effectiveness of the proposed method. The model achieves a MAP of 0.58 in large-scale lookup tasks, outperforming previous approaches by approximately 6%. In live song identification scenarios, the R-precision reaches 0.646, confirming robustness in noisy, real-world environments. These results highlight the importance of multi-pitch features and the novel loss function in capturing the subtle similarities between cover versions.
Beyond technical improvements, this work has significant implications for industry applications such as copyright enforcement, automated playlist curation, and live performance recognition. By enabling more accurate and scalable content matching, it paves the way for smarter music content management systems. Nevertheless, challenges remain, including handling extremely noisy recordings and scaling the approach for real-time deployment. Future research will focus on multimodal integration, dataset expansion, and computational efficiency, aiming to bring this promising technology closer to industrial adoption.
Deep Dive
Abstract
Automatic cover detection -- the task of finding in a audio dataset all covers of a query track -- has long been a challenging theoretical problem in MIR community. It also became a practical need for music composers societies requiring to detect automatically if an audio excerpt embeds musical content belonging to their catalog. In a recent work, we addressed this problem with a convolutional neural network mapping each track's dominant melody to an embedding vector, and trained to minimize cover pairs distance in the embeddings space, while maximizing it for non-covers. We showed in particular that training this model with enough works having five or more covers yields state-of-the-art results. This however does not reflect the realistic use case, where music catalogs typically contain works with zero or at most one or two covers. We thus introduce here a new test set incorporating these constraints, and propose two contributions to improve our model's accuracy under these stricter conditions: we replace dominant melody with multi-pitch representation as input data, and describe a novel prototypical triplet loss designed to improve covers clustering. We show that these changes improve results significantly for two concrete use cases, large dataset lookup and live songs identification.
References (20)
Cover Detection Using Dominant Melody Embeddings
G. Doras, Geoffroy Peeters
Cross recurrence quantification for cover song identification
J. Serrà, Xavier Serra, R. Andrzejak
Finding Cover Songs by Melodic Similarity Christian Sailer and
Fraunhofer Idmt Langewiesener
The song remains the same: identifying versions of the same piece using tonal descriptors
E. Gómez, P. Herrera
Multiple Fundamental Frequency Estimation by Summing Harmonic Amplitudes
Anssi Klapuri
Audio-Based Cover Song Retrieval Using Approximate Chord Sequences: Testing Shifts, Gaps, Swaps and Beats
J. Bello
One shot learning of simple visual concepts
B. Lake, R. Salakhutdinov, Jason Gross et al.
Large-Scale Cover Song Recognition Using the 2D Fourier Transform Magnitude
Thierry Bertin-Mahieux, D. Ellis
Siamese Neural Networks for One-Shot Image Recognition
Gregory R. Koch
Known Artist Live Song ID: A Hashprint Approach
T. Tsai, Thomas Prätzlich, Meinard Müller
Deep Salience Representations for F0 Estimation in Polyphonic Music
Rachel M. Bittner, Brian McFee, J. Salamon et al.
Neighbourhood Components Analysis
J. Goldberger, S. Roweis, Geoffrey E. Hinton et al.
Distance Metric Learning for Large Margin Nearest Neighbor Classification
Kilian Q. Weinberger, L. Saul
Identifying `Cover Songs' with Chroma Features and Dynamic Programming Beat Tracking
D. Ellis, Graham E. Poliner
Using the Similarity of Main Melodies to Identify Cover Versions of Popular Songs for Music Document Retrieval
Wei-Ho Tsai, Hung-Ming Yu, H. Wang
A Mid-Level Representation for Melody-Based Retrieval in Audio Collections
M. Marolt
Cover song detection: From high scores to general classification
Suman V. Ravuri, D. Ellis
Melody Extraction From Polyphonic Music Signals Using Pitch Contour Characteristics
J. Salamon, E. Gómez
Moving Beyond Feature Design: Deep Architectures and Automatic Feature Learning in Music Informatics
Eric J. Humphrey, J. Bello, Yann LeCun
Tonal representations for music retrieval: from version identification to query-by-humming
J. Salamon, J. Serrà, E. Gómez
Cited By (19)
Combining musical features for cover detection
Fully Fused Cover Song Identification Model via Feature Fusing and Clustering
Cover Song Identification in Practice with Multimodal Co-Training
Regularization-free Diffeomorphic Temporal Alignment Nets
Self-Supervised Learning of Multi-Level Audio Representations for Music Segmentation
Deep Learning for Audio and Music
Efficient Retrieval of Music Recordings Using Graph-Based Index Structures
An Educational Guide through the FMP Notebooks for Teaching and Learning Fundamentals of Music Processing
Detecting Cover Songs with Pitch Class Key-Invariant Networks
WideResNet with Joint Representation Learning and Data Augmentation for Cover Song Identification
Towards Robust Version Identification in the Wild: A Dataset, Benchmark, and Fine-Tuning Study
Training audio transformers for cover song identification
Two-Stage Pedestrian Detection Model Using a New Classification Head for Domain Generalization
A Semi-Supervised Deep Learning Approach to Dataset Collection for Query-By-Humming Task
Predict and Interpret Health Risk Using Ehr Through Typical Patients
Imputation with Inter-Series Information from Prototypes for Irregular Sampled Time Series
Discogs-VI: A Musical Version Identification Dataset Based on Public Editorial Metadata
Diffeomorphic Temporal Alignment Nets for Time-series Joint Alignment and Averaging
Imputation with Inter-Series Information from Prototypes for Healthcare Time Series