Accurate and Scalable Version Identification Using Musically-Motivated Embeddings
MOVE employs musically-motivated deep embeddings with triplet loss and hard mining, achieving state-of-the-art accuracy and scalability in music version identification.
Key Findings
Methodology
MOVE introduces a novel input feature—crema-PCP—combining it with multi-channel adaptive attention and standardized latent space. The architecture comprises five convolutional blocks with no padding to ensure transposition invariance, utilizing large receptive fields through dilation. Content summarization is achieved via a multi-channel attention mechanism and auto-pool, capturing long-term temporal features. Training employs triplet loss with online hard triplet mining, augmented by data augmentation strategies including pitch transposition, time stretching, and warping. The embeddings are normalized using non-parametric batch normalization, enabling efficient similarity measurement via Euclidean distance. This design balances high accuracy with scalability for large datasets.
Key Results
- On Da-TACOS, with 16k embedding dimensions, MAP reaches 0.507, outperforming all baselines; on YTC, MAP hits 0.888, setting new state-of-the-art. Ablation studies confirm that data augmentation, transposition invariance, and hard triplet mining significantly boost performance. The model scales efficiently, enabling fast retrieval in large music collections. Performance saturates around 16k dimensions, with 4k still surpassing previous methods, demonstrating effective trade-offs between complexity and accuracy.
Significance
This work advances music version identification by combining domain-specific representations with deep metric learning, addressing the challenge of robustness across diverse musical variations. The ability to perform accurate, scalable retrieval on large datasets impacts music industry applications such as copyright enforcement, music recommendation, and digital archiving. The approach also provides a framework for future multi-modal and multi-feature fusion, paving the way for more comprehensive music understanding systems.
Technical Contribution
The paper presents several innovations: introduction of crema-PCP as input, multi-channel adaptive attention for temporal summarization, expansion of receptive field via dilated convolutions, and normalization of embedding space through non-parametric batch normalization. The training leverages triplet loss with online hard triplet mining, ensuring robust separation of musical versions. The architecture supports high-dimensional embeddings, facilitating large-scale retrieval with Euclidean distance, and surpasses existing deep learning approaches in both accuracy and efficiency.
Novelty
This is the first application of crema-PCP combined with deep metric learning for music version identification. The multi-channel attention mechanism for temporal content summarization and the standardized latent space are novel contributions. Unlike prior feature-based or local alignment methods, MOVE learns a global, content-invariant embedding that generalizes across diverse musical variations, setting new benchmarks in both accuracy and scalability.
Limitations
- Despite high performance, the model struggles with extreme variations such as heavy editing or drastic key changes, due to limited training data covering such cases.
- High-dimensional embeddings (e.g., 16k) improve accuracy but increase computational load, limiting real-time deployment.
- Currently relies on single content representation; integrating multi-modal features like lyrics or rhythm could further enhance robustness.
Future Work
Future directions include integrating multi-modal data (lyrics, rhythm), optimizing network architecture for real-time applications, and exploring unsupervised or semi-supervised training to reduce reliance on labeled datasets. Additionally, investigating the learned similarity space could yield insights into musical relationships and version variations, broadening the scope of automated music understanding.
AI Executive Summary
Music version identification remains a critical challenge in music information retrieval, especially as digital music libraries grow exponentially. Traditional approaches, relying on feature matching or local alignment, often falter in scalability and robustness against variations such as key changes, tempo shifts, or editing. Recent deep learning methods have made strides by learning low-dimensional embeddings, but many still struggle with large-scale deployment and invariance to musical transformations.
This paper introduces MOVE, a novel deep metric learning framework that leverages musically-motivated embeddings for accurate and scalable version identification. Central to MOVE is the crema-PCP feature, which captures pitch class energies derived from an intermediate chord estimation model, providing a robust musical content representation. The architecture employs five convolutional blocks with no padding, enabling the model to learn transposition-invariant features while expanding the receptive field to encompass up to 30 seconds of content. To summarize temporal information effectively, a multi-channel adaptive attention mechanism combined with auto-pool is used, allowing the model to focus on salient content across long durations.
Training involves triplet loss with online hard triplet mining, which ensures the model learns to distinguish subtle differences between versions. Data augmentation strategies, including pitch transposition, time stretching, and warping, further improve robustness. The embeddings are normalized using non-parametric batch normalization, facilitating efficient similarity computation via Euclidean distance.
Experimental results on two public datasets, Da-TACOS and YouTubeCovers, demonstrate that MOVE outperforms existing state-of-the-art systems, achieving MAP scores of 0.507 and 0.888 respectively. Ablation studies confirm the importance of each component, with performance saturating around 16k embedding dimensions. The approach’s scalability and robustness make it suitable for large-scale music content management, copyright enforcement, and retrieval applications.
Looking ahead, future work will explore multi-modal fusion, real-time deployment, and deeper analysis of the learned similarity space, aiming to further enhance the system’s accuracy and applicability across diverse musical genres and variations.
Deep Analysis
Background
随着数字音乐的普及，海量音乐内容的管理和检索成为亟待解决的问题。传统方法多依赖特征匹配、局部对齐或哈希技术，效果在小规模数据集上尚可，但在大规模场景中存在效率低、鲁棒性差等问题。深度学习技术的发展带来了新的突破，卷积神经网络（CNN）和深度度量学习（如Triplet Loss）被引入音乐检索，提升了内容表达和相似性度量能力。代表性工作包括Xu等人的多尺度卷积网络和Doras与Peeters的主旋律嵌入方法。这些方法在小数据集上表现优异，但在大规模应用中仍面临性能和效率的挑战。近年来，伴随大规模音乐数据集（如Million Song Dataset）的出现，研究重点转向模型的扩展性和跨版本鲁棒性，推动深度嵌入技术快速发展。
Core Problem
音乐版本识别的核心难题在于如何在不同的演绎、调性、节奏甚至编辑变异中，准确识别出同一首歌曲的不同版本。传统方法多依赖局部特征匹配，易受变异影响，且难以支持大规模检索。深度学习虽能提取丰富内容特征，但在保证转调不变性、长时序信息捕获和高效检索方面仍存在瓶颈。如何设计一种既鲁棒又高效的模型，是当前亟待攻克的难题。
Innovation
本文的创新点包括：1）引入crema-PCP作为输入特征，结合中间层输出，增强内容表达的鲁棒性；2）设计多通道自适应注意机制，有效总结长时序内容，提升模型对变异的容错能力；3）利用无填充卷积扩大感受野，捕获长达30秒的内容信息；4）采用非参数批归一化，确保嵌入空间的统计标准化，便于距离度量；5）结合三元组损失和在线硬样本挖掘，提升模型区分不同版本的能力。这些创新共同实现了在准确性和大规模检索中的突破。
Methodology
- �� 输入：采用crema-PCP特征，提取每帧的能量值，构建12×T矩阵，经过数据增强（转调、时间拉伸、扭曲）以增强鲁棒性。
- �� 网络架构：由五个卷积块组成，第一层实现转调不变性（通过拼接两份输入，卷积后最大池化选择最高激活），后续层采用无填充卷积以扩大感受野，捕获长时序内容。
- �� 内容总结：引入多通道自适应注意机制，结合auto-pool实现对时间内容的加权总结，增强对变异的鲁棒性。
- �� 嵌入空间：在线性层后使用非参数批归一化，确保嵌入特征的统计标准化，方便距离计算。
- �� 训练：采用三元组损失，利用在线硬样本挖掘，从批次中挑选最难的正负样本，优化模型区分能力。
- �� 评估：在Da-TACOS和YTC两个公开数据集上测试，指标包括MAP和第一相关结果的平均排名，验证模型优越性。
Experiments
使用97,905首歌曲组成的私有数据集进行训练，划分为训练和验证集。采用多样的数据增强策略，包括转调、时间拉伸和扭曲，增强模型鲁棒性。模型在不同嵌入维度（128到32k）上进行调优，发现16k维效果最佳。消融实验验证了数据增强、转调不变架构和硬样本挖掘的重要性。与现有方法相比，MOVE在MAP指标上提升超过10%，在大规模数据集上表现优异。模型的长时序感知能力显著优于传统特征匹配方法，验证了内容总结和感受野扩展的有效性。
Results
在Da-TACOS上，16k嵌入的MAP达0.507，优于所有对比方法；YTC上，MAP达0.888，刷新了最优记录。消融实验显示，缺少数据增强或硬样本挖掘会导致性能下降约20%。模型支持大规模快速检索，利用欧氏距离实现高效匹配。长时序内容捕获能力显著优于传统特征，验证了模型设计的有效性。
Applications
该模型适用于音乐版权保护、内容管理、音乐推荐和检索系统。只需提供音频，便能实现高效准确的版本识别，支持大规模数据库的快速检索。未来可结合歌词、节奏等多模态信息，提升识别鲁棒性，推动音乐产业智能化发展。
Limitations & Outlook
模型在极端变异（如大幅调性变化或复杂编辑）下表现仍有限，主要因训练数据不足。高维嵌入虽提升性能，但计算成本较高，限制实时应用。当前主要依赖单一内容特征，未来需融合多模态信息以增强鲁棒性。
Plain Language Accessible to non-experts
想象你在一家工厂里，每天都在生产不同的商品。有时候商品会被重新包装或改装，但工厂需要一种方法，能快速识别出这些不同包装但其实是同一件商品的“指纹”。传统的方法就像用标签识别，但标签可能被换掉或改动。现在，工厂引入了一种智能检测系统，它通过分析商品的内部结构和特征，学习到一种“指纹”，可以在不同包装中找到相似的“指纹”。这个系统用深度学习训练，能在海量商品中快速找到相似的商品，不管它们的包装如何变化。这样，工厂就能更高效地管理库存，防止假冒伪劣商品流入市场。这个系统的核心在于它能学习到商品的“深层特征”，而不是表面标签，从而实现准确识别。
ELI14 Explained like you're 14
想象你在学校，有很多不同版本的同一首歌，比如有人唱得快，有人唱得慢，甚至调调也不一样。你要找到这些不同版本的歌，就像找朋友一样，听一听，看看是不是同一首。这就像用耳朵去识别不同的版本，但如果差别太大，光靠耳朵很难分辨。科学家们发明了一种聪明的办法，让电脑也能学会这个技能。他们教电脑用一种特殊的“音乐指纹”来记住每首歌的特点，就像用指纹识别自己。这个“指纹”可以捕捉到歌曲的核心内容，不管它变了多少调、速度或加入了新歌词。通过训练，电脑变得越来越聪明，能在几百万首歌中快速找到相似的版本。这样，不仅可以帮人们整理音乐库，还能保护版权，防止盗版。就像你用一个超级厉害的搜索引擎，找到所有相似的歌一样，科学家们让电脑变成了这个“音乐侦探”。
Abstract
The version identification (VI) task deals with the automatic detection of recordings that correspond to the same underlying musical piece. Despite many efforts, VI is still an open problem, with much room for improvement, specially with regard to combining accuracy and scalability. In this paper, we present MOVE, a musically-motivated method for accurate and scalable version identification. MOVE achieves state-of-the-art performance on two publicly-available benchmark sets by learning scalable embeddings in an Euclidean distance space, using a triplet loss and a hard triplet mining strategy. It improves over previous work by employing an alternative input representation, and introducing a novel technique for temporal content summarization, a standardized latent space, and a data augmentation strategy specifically designed for VI. In addition to the main results, we perform an ablation study to highlight the importance of our design choices, and study the relation between embedding dimensionality and model performance.
References (20)
Da-TACOS: A Dataset for Cover Song Identification and Understanding
Furkan Yesiler, C. Tralie, A. Correya et al.
Multimodal similarity between musical streams for cover version detection
Rémi Foucard, Jean-Louis Durrieu, Mathieu Lagrange et al.
The song remains the same: identifying versions of the same piece using tonal descriptors
E. Gómez, P. Herrera
A Mid-level Melody-based Representation for Calculating Audio Similarity
M. Marolt
Unsupervised Detection of Cover Song Sets: Accuracy Improvement and Original Identification
J. Serrà, M. Zanin, C. Laurier et al.
Identification of versions of the same musical composition by processing audio descriptions
Joan Serrà Julià
The Million Song Dataset
Thierry Bertin-Mahieux, D. Ellis, B. Whitman et al.
Large-Scale Cover Song Recognition Using the 2D Fourier Transform Magnitude
Thierry Bertin-Mahieux, D. Ellis
Music Shapelets for Fast Cover Song Recognition
Diego Furtado Silva, V. Souza, Gustavo E. A. P. A. Batista
Known Artist Live Song ID: A Hashprint Approach
T. Tsai, Thomas Prätzlich, Meinard Müller
SONG IDENTIFICATION WITH 2 D FOURIER TRANSFORM SEQUENCES
Prem Seetharaman
Structured Training for Large-Vocabulary Chord Recognition
Brian McFee, J. Bello
A Review of Audio Fingerprinting
Pedro Cano, Eloi Batlle, T. Kalker et al.
Identifying `Cover Songs' with Chroma Features and Dynamic Programming Beat Tracking
D. Ellis, Graham E. Poliner
Efficient Index-Based Audio Matching
F. Kurth, Meinard Müller
Cross recurrence quantification for cover song identification
J. Serrà, Xavier Serra, R. Andrzejak
Temporal Pyramid Pooling Convolutional Neural Network for Cover Song Identification
Zhesong Yu, Xiaoshuo Xu, Xiaoou Chen et al.
Audio Content-Based Music Retrieval
Peter Grosche, Meinard Müller, J. Serrà
Data Driven and Discriminative Projections for Large-Scale Cover Song Identification
Eric J. Humphrey, Oriol Nieto, J. Bello
FaceNet: A unified embedding for face recognition and clustering
Florian Schroff, Dmitry Kalenichenko, James Philbin
Cited By (20)
A music similarity function based on probabilistic linear discriminant analysis for cover song identification
Audio-Based Musical Version Identification: Elements and challenges
Assessing Algorithmic Biases for Musical Version Identification
On the Robustness of Cover Version Identification Models: A Study Using Cover Versions from YouTube
The Words Remain the Same: Cover Detection with Lyrics Transcription
Detecting Cover Songs with Pitch Class Key-Invariant Networks
MulKINet: Multi-Stage Key-Invariant Convolutional Neural Networks for Accurate and Fast Cover Song Identification
Less is more: Faster and better music version identification with embedding distillation
Combining musical features for cover detection
Fully Fused Cover Song Identification Model via Feature Fusing and Clustering
And what if two musical versions don't share melody, harmony, rhythm, or lyrics ?
Bytecover: Cover Song Identification Via Multi-Loss Training
Efficient Retrieval of Music Recordings Using Graph-Based Index Structures
LEARNING AUDIO EMBEDDINGS VIA LYRICS ALIGNMENT FOR SCALABLE VERSION IDENTIFICATION 2025
Investigating the Efficacy of Music Version Retrieval Systems for Setlist Identification
Cover Song Identification in Practice with Multimodal Co-Training
Music Version Retrieval from YouTube: How to Formulate Effective Search Queries?
Artist Similarity for Everyone: A Graph Neural Network Approach
Using Weakly Aligned Score-Audio Pairs to Train Deep Chroma Models for Cross-Modal Music Retrieval
Learning Sound Representations Using Triplet-loss