Towards Robust Version Identification in the Wild: A Dataset, Benchmark, and Fine-Tuning Study
Introduces DiVers dataset with over 1.1 million music versions, enhancing robustness in real-world scenarios, significantly improving music version identification (VI) performance.
Key Findings
Methodology
This study employs a deep neural network-based music version identification framework, leveraging the large-scale DiVers dataset that combines official (DVI) and in-the-wild (YVI) content. The input features are based on Constant-Q Transform (CQT), with a hop size of 20 ms, spanning 7 octaves, and normalized for training. The model architecture builds upon the DVINet+ backbone, scaled up to 20.7 million parameters, trained using Triplet Loss with hard triplet mining, and enhanced with data augmentation techniques like SpecAugment. The dataset incorporates automatically generated tags (e.g., cover, live, karaoke) and segment-level music/non-music predictions via PANN. Training involves from-scratch learning and fine-tuning, evaluated with metrics such as MAP and NAR across multiple datasets, including DVI, SHS100K, and in-the-wild collections, to validate robustness against noise and content variability.
Key Results
- Models trained on DiVers exhibit a 5-8% increase in MAP and a 1-3 point decrease in NAR in noisy environments compared to models trained solely on official datasets. The fine-tuned CLEWS model achieves MAP up to 0.708 and NAR as low as 9.61 across diverse datasets, demonstrating enhanced robustness.
- Expanding training data with DiVers-L yields consistent performance improvements across all evaluation metrics and datasets, especially in scenarios involving non-professional recordings, live performances, and reaction videos, confirming the importance of data diversity.
- The large-scale dataset enables models to generalize better in real-world conditions, with significant gains in in-the-wild and noisy content, thus bridging the gap between academic benchmarks and practical applications.
Significance
This work addresses a critical bottleneck in music information retrieval: the difficulty of accurately recognizing different versions of a musical work in uncontrolled, noisy environments typical of online platforms like YouTube. By constructing the largest and most diverse dataset to date, DiVers, and demonstrating its effectiveness through comprehensive evaluation, the study paves the way for more robust, scalable, and real-world applicable music recognition systems. Such advancements can revolutionize copyright enforcement, personalized music recommendation, and live performance analysis, making automated systems more resilient to content variability and environmental noise. The integration of automatic tags and segment-level predictions further enhances content understanding, enabling nuanced applications like genre-specific retrieval and contextual content analysis.
Technical Contribution
The paper’s key technical innovations include: 1) the creation of DiVers, a massive, heterogeneous dataset combining official and in-the-wild content, annotated with automatic tags and segment-level predictions; 2) the adaptation of deep metric learning with Triplet Loss to improve discriminative power across diverse versions; 3) the systematic evaluation of training strategies, including from-scratch training and fine-tuning, demonstrating how data scale and diversity influence model robustness; 4) the integration of automatic tag-based stratification and segment-level analysis to understand model behavior under different content conditions. These contributions collectively advance the state-of-the-art in robust music version recognition.
Novelty
This is the first work to assemble such a large-scale, diverse dataset that explicitly includes user-generated, unofficial, and noisy content for music version identification. Unlike prior datasets limited to curated, high-quality recordings, DiVers captures the complexity of real-world scenarios. The automatic tagging and segment-level detection mechanisms further differentiate this work by providing richer content annotations, enabling more nuanced model training and evaluation. The combination of large-scale data, automatic content descriptors, and rigorous evaluation across multiple environments marks a significant step forward in bridging the gap between lab conditions and practical applications.
Limitations
- Despite its scale, the dataset relies on automatic tags, which may contain inaccuracies, potentially affecting model training and evaluation. Improving label quality remains an ongoing challenge.
- Training large models like DVINetX and fine-tuning CLEWS requires substantial computational resources, limiting accessibility for some research groups.
- The models still face challenges in extreme noise conditions or highly complex content scenarios, such as overlapping speech or multiple concurrent performances, indicating room for further robustness improvements.
Future Work
Future directions include integrating multimodal data such as video and lyrics to enhance recognition accuracy, developing unsupervised or semi-supervised learning approaches to reduce reliance on automatic labels, and optimizing model architectures for deployment on resource-constrained devices. Additionally, expanding the dataset to include more cultural and linguistic diversity will improve global applicability. Exploring zero-shot learning and domain adaptation techniques could further enhance robustness in unseen environments, ultimately bringing music recognition technology closer to seamless real-world deployment.
AI Executive Summary
Music version identification (VI) is a fundamental task in music information retrieval, aiming to recognize different renditions of the same musical work. Traditional approaches relied heavily on handcrafted features and curated datasets, which limited their effectiveness in real-world scenarios characterized by noisy, user-generated content. With the rise of online platforms like YouTube, the diversity and complexity of music content have increased dramatically, posing new challenges for VI systems. Variations in audio quality, environmental noise, unofficial recordings, and non-musical content such as commentary or reaction videos make the task significantly harder.
To address these challenges, this study introduces DiVers, a comprehensive dataset comprising over 1.1 million music versions, integrating official (DVI) and in-the-wild (YVI) content. The dataset is enriched with automatically assigned tags—cover, live, karaoke, tutorial—and segment-level predictions of music presence, enabling models to learn richer content representations. The authors employ a deep convolutional neural network architecture based on DVINet+, trained with Triplet Loss, and further enhance robustness through fine-tuning on in-the-wild data. The training process incorporates extensive data augmentation and rigorous evaluation metrics such as MAP and NAR.
Experimental results demonstrate that models trained on DiVers outperform baseline systems trained on traditional datasets, showing a 5-8% increase in MAP and a 1-3 point reduction in NAR in noisy environments. Fine-tuning further boosts performance, especially on in-the-wild content, with MAP reaching 0.708 and NAR dropping below 10 in some cases. These improvements confirm that large-scale, diverse datasets significantly enhance the robustness and generalization of VI systems.
The implications of this work are profound. By bridging the gap between controlled, curated datasets and the chaotic reality of user-generated content, the proposed approach enables more reliable music recognition in practical applications such as copyright enforcement, personalized recommendations, and live event analysis. The automatic tagging and segment detection mechanisms also facilitate content stratification, supporting more nuanced retrieval and analysis.
Despite these advances, challenges remain. The reliance on automatic labels introduces some uncertainty, and the high computational cost of training large models limits accessibility. Future research directions include multimodal integration, unsupervised learning, and domain adaptation to further improve robustness and scalability. Overall, this work marks a significant step toward deploying music recognition technology in real-world, noisy environments, promising a future where machines can understand and organize the vast diversity of musical content online with unprecedented accuracy.
Deep Dive
Glossary
Constant-Q Transform (CQT)
一种频谱分析方法，具有对数尺度的频率分辨率，适合音乐信号的特征提取。在论文中用于输入特征的构建。
作为模型的输入特征，用于捕捉音乐的调性和谐波结构。
Triplet Loss (三元组损失)
一种深度学习中的距离学习损失函数，旨在拉近同一类别样本的特征距离，推远不同类别样本的距离。在模型训练中用于增强区分能力。
用于训练深度模型，使不同版本的音乐在特征空间中具有更好的可区分性。
SpecAugment
一种数据增强技术，通过对频谱图进行时间和频率的随机遮挡，提高模型的泛化能力。
在训练过程中应用于输入特征，增强模型对噪声和变异的鲁棒性。
MAP (Mean Average Precision)
一种信息检索中的评价指标，衡量模型在所有查询中的平均排序准确率。
用于评估音乐版本识别系统的整体检索性能。
NAR (Normalized Average Rank)
衡量正确匹配在排序中的平均排名，值越低表示性能越好。
补充MAP，用于评估模型的排名质量。
DVI (Discogs-VI)
基于Discogs元数据构建的官方音乐版本数据集，包含近50万版本。
作为训练和评估的基础数据集之一。
YVI (YouTube In-the-Wild Versions)
在YouTube平台上采集的非官方、用户生成的音乐版本，丰富了内容多样性。
补充DVI，增强模型在真实环境中的适应性。
DiVers (Diverse Versions)
本文提出的超大规模、多样化音乐版本数据集，结合官方和在野内容，超过110万版本。
研究的核心数据资源，显著提升模型鲁棒性。
Fine-tuning (微调)
在预训练模型基础上，利用特定数据集进行再训练，以适应特定任务或环境。
用于提升模型在噪声环境中的表现。
Segment-level prediction (段级预测)
对音频片段进行音乐/非音乐的分类，提供内容的局部信息。
增强模型对内容属性的理解和分析能力。
Auto-tagging (自动标签)
利用模型自动为音乐内容赋予类别标签，如风格、乐器、现场等。
丰富内容描述，支持内容分析和检索。
Sound event classification (声音事件分类)
识别音频中的不同声音事件，如音乐、讲话、环境噪声等。
在段级检测中应用，区分音乐与非音乐内容。
Embedding space (嵌入空间)
高维特征空间，用于表示音频样本的语义特征。
分析模型在训练前后特征空间的变化，理解模型鲁棒性。
Data augmentation (数据增强)
通过对训练数据进行变换（如时间扭曲、频谱遮挡）增加多样性，提升模型泛化能力。
在训练过程中广泛应用于提升模型鲁棒性。
In-the-wild content (野生内容)
指在非受控环境下采集的内容，如用户上传的音乐视频。
模型需要在此类内容中表现出良好的识别能力。
Open Questions Unanswered questions from this research
- 1 尽管DiVers大幅提升了模型鲁棒性，但自动标签的准确性仍有限，未来需要引入半监督或无监督学习方法以减少标签偏差，提升模型在极端环境下的表现。
- 2 模型训练成本高，尤其是在大规模数据集上，限制了其在资源有限环境中的应用。未来应探索模型压缩和加速技术。
- 3 当前模型在极端噪声和复杂内容（如多重演奏、背景噪声叠加）场景下仍存在识别失误，未来需结合多模态信息（如视频、歌词）提升鲁棒性。
- 4 跨文化和多语言环境下的版本识别能力尚未充分验证，未来应扩展多语种、多文化数据，增强模型的普适性。
- 5 缺乏端到端的实时识别系统，未来需结合边缘计算和在线学习技术，实现实时、低延迟的版本识别。
Applications
Immediate Applications
版权监测
利用模型自动识别音乐作品的不同版本，帮助版权方监控网络上传播的侵权内容，自动检测未经授权的使用，提升版权保护效率。
音乐推荐
通过准确识别用户偏好的不同版本，提升个性化推荐的准确性和多样性，增强用户体验。
现场演出分析
在现场或录制演出中快速识别演奏版本，辅助音乐会管理和内容归档，为演出分析提供技术支持。
Long-term Vision
多模态内容理解
结合视频、歌词、演唱者信息，构建更全面的音乐内容理解模型，实现跨模态的智能检索和内容分析。
无监督学习与自适应系统
发展无需大量标注的模型，适应不断变化的内容环境，实现持续学习和自我优化，推动音乐识别技术的普及与应用。
Abstract
Existing datasets for musical version identification (VI) are primarily derived from curated metadata sources such as SecondHandSongs and Discogs, and are therefore dominated by professionally recorded tracks. This leads to a domain mismatch with real-world scenarios, where amateur and user-generated content is prevalent. To address this limitation, we introduce DiVers, a large-scale VI dataset comprising over 1.1 million musical versions, with train-validation-test splits compatible with established datasets such as Discogs-VI-YT, SHS100K, and Da-TACOS. In addition to standard version-level annotations, DiVers provides automatically assigned tags (e.g., instrumental, live) and segment-level predictions indicating the presence or absence of music. We evaluate the proposed dataset by training state-of-the-art VI systems. Our results show that models trained on DiVers achieve substantially improved robustness to acoustically diverse and noisy inputs, while maintaining a stable performance on cleaner, studio-quality benchmarks. We release the dataset metadata, code for its construction, and all experimental pipelines to support reproducibility.
References (20)
Learning a Representation for Cover Song Identification Using Convolutional Neural Network
Zhesong Yu, Xiaoshuo Xu, Xiaoou Chen et al.
Discogs-VI: A Musical Version Identification Dataset Based on Public Editorial Metadata
R. O. Araz, Xavier Serra, D. Bogdanov
On the Robustness of Cover Version Identification Models: A Study Using Cover Versions from YouTube
Simon Hachmeier, Robert Jäschke
Supervised contrastive learning from weakly-labeled audio segments for musical version matching
Joan Serrà, R. O. Araz, Dmitry Bogdanov et al.
A Benchmark and Robustness Study of In-Context-Learning with Large Language Models in Music Entity Detection
Simon Hachmeier, Robert Jäschke
Evaluation of Deep Audio Representations for Semantic Sound Similarity
R. O. Araz, Dmitry Bogdanov, Pablo Alonso-Jiménez et al.
Leveraging User-Generated Metadata of Online Videos for Cover Song Identification
Simon Hachmeier, Robert Jäschke
Bytecover3: Accurate Cover Song Identification On Short Queries
Xingjian Du, Zijie Wang, Xia Liang et al.
Popular music reaction videos: Reactivity, creator labor, and the performance of listening online
Byrd Mcdaniel
How Robust are Audio Embeddings for Polyphonic Sound Event Tagging?
J. Abeßer, S. Grollmisch, Meinard Müller
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
Daniel S. Park, William Chan, Yu Zhang et al.
Accurate and Scalable Version Identification Using Musically-Motivated Embeddings
Furkan Yesiler, J. Serrà, Emilia G'omez
Leveraging Whisper Embeddings for Audio-based Lyrics Matching
Eleonora Mancini, Joan Serrà, Paolo Torroni et al.
Evaluation of Algorithms Using Games: The Case of Music Tagging
Edith Law, Kris West, Michael I. Mandel et al.
A Prototypical Triplet Loss for Cover Detection
G. Doras, Geoffroy Peeters
Identifying `Cover Songs' with Chroma Features and Dynamic Programming Beat Tracking
D. Ellis, Graham E. Poliner
Da-TACOS: A Dataset for Cover Song Identification and Understanding
Furkan Yesiler, C. Tralie, A. Correya et al.
OpenAI GPT-5 System Card
Aaditya K. Singh, A. Fry, Adam Perelman et al.
PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition
Qiuqiang Kong, Yin Cao, Turab Iqbal et al.
And what if two musical versions don't share melody, harmony, rhythm, or lyrics ?
M. Abrassart, G. Doras