Moonbeam: A MIDI Foundation Model Using Both Absolute and Relative Music Attributes
Moonbeam model combines absolute and relative music attributes to enhance MIDI music generation and understanding.
Key Findings
Methodology
The Moonbeam model employs a Transformer architecture with Multidimensional Relative Attention (MRA) and a novel tokenization method to effectively capture both absolute and relative music information. Pretrained on 81.6K hours of MIDI data, the model shows strong generalization capabilities across various music tasks.
Key Results
- Moonbeam outperforms other large-scale pretrained music models in accuracy and F1 score across three music classification tasks on four datasets.
- In conditional music generation tasks, Moonbeam surpasses a strong Transformer baseline with a REMI-like tokenizer.
- Ablation studies reveal that MRA and the GRU decoder significantly reduce test perplexity.
Significance
The Moonbeam model holds significant implications for music generation and understanding, particularly in handling multi-track, multi-style MIDI data. Its innovative tokenization method and relative attention mechanism provide new insights for AI research in the music domain.
Technical Contribution
Moonbeam introduces a Multidimensional Relative Attention mechanism that captures relative music information without additional parameters. Its tokenization method supports various MIDI data forms, demonstrating strong generalization capabilities.
Novelty
Moonbeam is the first to pretrain on a large, diverse MIDI dataset, combining absolute and relative music attributes to enhance model generation and understanding capabilities.
Limitations
- The model may face computational resource constraints when handling extremely long sequences.
- Performance on specific music styles may not match specialized models.
Future Work
Future research could explore further optimization of the MRA mechanism or integration with other music data formats like audio to enhance model performance.
AI Executive Summary
The Moonbeam model is a Transformer-based music foundation model designed specifically for handling MIDI data. By combining absolute and relative music attributes, it offers more precise music generation and understanding capabilities.
The model employs an innovative Multidimensional Relative Attention mechanism and tokenization method, enabling it to process various forms of MIDI data, including single-track and multi-track music. Pretrained on 81.6K hours of MIDI data, Moonbeam demonstrates superior performance across multiple music tasks.
Experimental results show that Moonbeam outperforms existing pretrained models in several music classification tasks and excels in conditional music generation tasks. However, the model still faces challenges in computational resources when handling extremely long sequences, and future research could further optimize its attention mechanism.
Deep Analysis
Background
In recent years, AI foundation models have made significant progress across various domains. In music, although some symbolic music pretrained models exist, most are based on ABC format, limiting their generative capabilities. MIDI format, known for expressing performed music, has gained attention.
Core Problem
Existing symbolic music models face challenges in handling multi-track, multi-style MIDI data, particularly in tokenization and capturing relative music attributes. Effectively processing this data without increasing computational complexity is a key issue.
Innovation
Moonbeam introduces a Multidimensional Relative Attention mechanism and a novel tokenization method to capture both absolute and relative music attributes. This enables the model to handle various forms of MIDI data and perform well in generation and understanding tasks.
Methodology
- �� Utilize Transformer architecture for pretraining
- �� Introduce Multidimensional Relative Attention to capture relative music information
- �� Employ a novel tokenization method for diverse MIDI data
- �� Pretrain on 81.6K hours of MIDI data for large-scale learning
Experiments
Experiments were conducted on four datasets, covering three music classification tasks. The model was compared against a strong Transformer baseline in conditional music generation tasks, using accuracy and F1 score as evaluation metrics.
Results
Moonbeam excels in music classification tasks, surpassing other pretrained models in accuracy and F1 score. In conditional music generation tasks, the model also outperforms a baseline Transformer with a REMI-like tokenizer.
Applications
Moonbeam can be used for music generation, music classification, and music understanding tasks, particularly in scenarios requiring multi-track, multi-style MIDI data processing.
Limitations & Outlook
Despite Moonbeam's strong performance in various tasks, it faces computational resource challenges when handling extremely long sequences. Additionally, its performance on specific music styles may not match specialized models.
Plain Language Accessible to non-experts
Imagine you're in a kitchen cooking. Moonbeam is like a smart chef who knows the absolute taste of each ingredient (absolute music attributes) and can adjust the proportions based on different recipes (relative music attributes). This way, it can quickly create delicious dishes (generate music) based on different needs. Moreover, it can flexibly adjust recipes based on ingredient changes (music understanding).
ELI14 Explained like you're 14
Imagine you're playing a music game, and Moonbeam is like a super helper. It not only helps you recognize the exact position of each note but also helps you create new music pieces based on the rhythm and style. Just like in a game, you can choose different tools for different challenges, Moonbeam can also adjust strategies based on different music tasks. Isn't that cool?
Glossary
Transformer
A neural network architecture for processing sequential data, widely used in NLP and music generation.
The foundational architecture of the Moonbeam model.
MIDI
A standard format for storing and transmitting music information, capable of representing performed music.
The primary data source for the Moonbeam model.
Multidimensional Relative Attention
An innovative attention mechanism that captures relative music information without additional parameters.
A key mechanism in Moonbeam for capturing relative music attributes.
Tokenization
The process of converting input data into a sequence of tokens that the model can process.
A crucial step in Moonbeam for handling various MIDI data.
F1 Score
A metric for evaluating classification model performance, combining precision and recall.
Used to evaluate Moonbeam's performance in music classification tasks.
Open Questions Unanswered questions from this research
- 1 How to further optimize the MRA mechanism without increasing computational complexity?
- 2 How to effectively utilize computational resources when handling extremely long sequences?
Applications
Immediate Applications
Music Generation
Music creators can use Moonbeam to generate multi-track music, enhancing creative efficiency.
Long-term Vision
Music Education
By analyzing and generating music, it helps students better understand music theory and composition techniques.
Abstract
Moonbeam is a transformer-based foundation model for symbolic music, pretrained on a large and diverse collection of MIDI data totaling 81.6K hours of music and 18 billion tokens. Moonbeam incorporates music-domain inductive biases by capturing both absolute and relative musical attributes through the introduction of a novel domain-knowledge-inspired tokenization method and Multidimensional Relative Attention (MRA), which captures relative music information without additional trainable parameters. Leveraging the pretrained Moonbeam, we propose 2 finetuning architectures with full anticipatory capabilities, targeting 2 categories of downstream tasks: symbolic music understanding and conditional music generation (including music infilling). Our model outperforms other large-scale pretrained music models in most cases in terms of accuracy and F1 score across 3 downstream music classification tasks on 4 datasets. Moreover, our finetuned conditional music generation model outperforms a strong transformer baseline with a REMI-like tokenizer. We open-source the code, pretrained model, and generated samples on Github.