On the de-duplication of the Lakh MIDI dataset
Used contrastive learning BERT model to deduplicate Lakh MIDI dataset, filtering out at least 38,134 samples.
Key Findings
Methodology
This study employs a contrastive learning-based BERT model with various data augmentations to detect duplicates in the Lakh MIDI dataset. Using LMD-clean as a benchmark test set, rule-based methods and existing symbolic music retrieval models were evaluated, resulting in three different versions of the filtered list.
Key Results
- Filtered out 38,134 samples in the most conservative settings, accounting for 21.4% of 178,561 files.
- CLaMP3 model achieved the best nDCG and MRR scores at 0.697 and 0.709, respectively.
- CAugBERT model excelled in classification tasks, achieving the highest F1 score.
Significance
This research is significant in the music information retrieval field, addressing dataset duplication issues and improving model training reliability. By removing duplicates, it prevents data leakage between training, validation, and test sets, enhancing evaluation accuracy.
Technical Contribution
Introduced a BERT model combining contrastive learning and various data augmentations, effectively detecting duplicates in large-scale symbolic music datasets. This method outperforms existing approaches, especially in handling hard duplicates.
Novelty
First application of contrastive learning BERT model for dataset deduplication in symbolic music, offering a more efficient solution compared to traditional rule-based methods.
Limitations
- Challenges in detecting soft duplicates, particularly those with different arrangement styles.
- Requires significant computational resources for model training.
Future Work
Future work could explore more complex data augmentation strategies to improve soft duplicate detection accuracy and validate this method's generalizability on other symbolic music datasets.
AI Executive Summary
In the symbolic music domain, large-scale datasets are crucial for training deep learning models. However, datasets often contain duplicates due to web scraping. The Lakh MIDI dataset is one of the largest public resources in this field, yet its duplication issues have not been adequately addressed.
This study proposes an effective deduplication solution using a contrastive learning-based BERT model with various data augmentations. Multiple deduplication methods were evaluated on the LMD-clean benchmark test set, including rule-based methods and existing symbolic music retrieval models. Results show that the CAugBERT model excels in deduplication tasks, particularly in handling hard duplicates.
By removing duplicates, this study enhances the reliability of model training and evaluation, preventing inaccuracies caused by data leakage. This research is significant not only academically but also provides new insights for practical applications in the music information retrieval field.
Deep Analysis
Background
As data-driven methods become mainstream, large-scale datasets are increasingly important for training deep learning models. The Lakh MIDI dataset is a key resource in the symbolic music domain, but its duplication issues have not been fully addressed. These duplicates may arise from multiple user arrangements or simple metadata edits.
Core Problem
Dataset duplication leads to data leakage between training, validation, and test sets, affecting model evaluation accuracy. In music generation, subjective evaluation is costly, and objective metrics often fail to capture the quality of generated music.
Innovation
This study is the first to apply a contrastive learning BERT model for dataset deduplication in symbolic music. By using various data augmentations, it improves duplicate detection accuracy, especially for hard duplicates.
Methodology
- �� Use LMD-clean as a benchmark test set
- �� Evaluate rule-based methods and existing symbolic music retrieval models
- �� Train contrastive learning BERT model with various data augmentations
- �� Propose three different versions of the filtered list
Experiments
Experiments used LMD-clean as a benchmark test set, evaluating various deduplication methods, including rule-based methods and existing symbolic music retrieval models. Key metrics include nDCG and MRR to assess duplicate detection capability.
Results
CAugBERT model excelled in classification tasks, achieving the highest F1 score. CLaMP3 model achieved the best nDCG and MRR scores at 0.697 and 0.709, respectively.
Applications
The deduplication method can improve the training and evaluation reliability of symbolic music generation models, preventing inaccuracies caused by data leakage.
Limitations & Outlook
Challenges in detecting soft duplicates, particularly those with different arrangement styles. Requires significant computational resources for model training.
Plain Language Accessible to non-experts
Imagine a music library with many duplicate scores. These scores may be due to different people arranging the same song differently or simply modifying some details. Our task is to find these duplicate scores and remove them from the library. We use a smart method, similar to tagging each score with a unique label, and then use these labels to identify which scores are duplicates.
ELI14 Explained like you're 14
Imagine you have a music library with lots of duplicate songs. These songs might be because different people arranged the same song differently or just tweaked some details. Our job is to find these duplicates and remove them from the library. We use a clever method, like tagging each song with a unique label, and then use these labels to spot which songs are duplicates.
Glossary
Contrastive Learning
A machine learning method that learns data representations by comparing similarities between samples.
Used to train the BERT model for duplicate detection.
BERT
A pre-trained model for natural language processing that captures contextual information.
Used for duplicate detection in symbolic music datasets.
Lakh MIDI Dataset
A large symbolic music dataset containing numerous MIDI files.
Used in the study to evaluate deduplication methods.
nDCG
A metric for evaluating the performance of retrieval systems, considering relevance and ranking.
Used to assess the effectiveness of deduplication methods.
MRR
Mean Reciprocal Rank, used to evaluate the accuracy of retrieval systems.
Used to assess the effectiveness of deduplication methods.
Open Questions Unanswered questions from this research
- 1 How to effectively detect soft duplicates, especially those with different arrangement styles, remains a challenge.
Applications
Immediate Applications
Symbolic Music Generation
Improves the training and evaluation reliability of generation models, preventing inaccuracies caused by data leakage.
Long-term Vision
Music Information Retrieval
Provides new insights for practical applications in the music information retrieval field, enhancing system performance.
Abstract
A large-scale dataset is essential for training a well-generalized deep-learning model. Most such datasets are collected via scraping from various internet sources, inevitably introducing duplicated data. In the symbolic music domain, these duplicates often come from multiple user arrangements and metadata changes after simple editing. However, despite critical issues such as unreliable training evaluation from data leakage during random splitting, dataset duplication has not been extensively addressed in the MIR community. This study investigates the dataset duplication issues regarding Lakh MIDI Dataset (LMD), one of the largest publicly available sources in the symbolic music domain. To find and evaluate the best retrieval method for duplicated data, we employed the Clean MIDI subset of the LMD as a benchmark test set, in which different versions of the same songs are grouped together. We first evaluated rule-based approaches and previous symbolic music retrieval models for de-duplication and also investigated with a contrastive learning-based BERT model with various augmentations to find duplicate files. As a result, we propose three different versions of the filtered list of LMD, which filters out at least 38,134 samples in the most conservative settings among 178,561 files.