Generating Symbolic Music from Natural Language Prompts using an LLM-Enhanced Dataset
Generates symbolic music using LLM-enhanced MetaScore dataset, providing text and tag controls.
Key Findings
Methodology
The study uses a pretrained large language model to generate pseudo-natural language music descriptions, training text and tag-controlled music generation models using the MetaScore dataset. Models include MetaScore Transformer-Text and MetaScore Transformer-Tags.
Key Results
- In listening tests, both text and tag models outperform baseline models, with text models excelling in natural language interaction.
- Compared to Text2MIDI, MST-Text matches ground truth closely in pitch class entropy and scale consistency.
- Tag models excel in coherence and arrangement, while text models score highest in overall quality.
Significance
This research significantly enhances symbolic music generation's controllability and natural language interaction capabilities, advancing the field through MetaScore dataset and LLM-generated pseudo-natural language descriptions.
Technical Contribution
Introduces a new music representation method supporting multiple music attribute controls, achieving high-quality text-to-music generation through LLM-generated natural language descriptions.
Novelty
First to use LLM-generated music descriptions combined with MetaScore dataset for symbolic music generation, offering a more natural user interaction interface.
Limitations
- Model struggles with complex free-text inputs, potentially degrading music quality.
- Auto-generated tags in the dataset may lack accuracy, affecting tag model performance.
Future Work
Future work could explore handling complex free-text inputs and improving auto-tag generation accuracy.
AI Executive Summary
Recent advancements in audio-domain text-to-music generation have been significant, yet symbolic-domain music generation lags due to the lack of large-scale datasets. The MetaScore dataset, enhanced by LLM-generated pseudo-natural language descriptions, offers rich music metadata. We propose two models supporting text and tag-controlled music generation. Experimental results show these models perform excellently in listening tests, with text models offering superior natural language interaction. Despite this, models face limitations in handling complex text inputs, and future research aims to improve these aspects. The MetaScore dataset and its generation models open new possibilities in symbolic music generation.
Deep Analysis
Background
Symbolic music generation systems produce editable music formats, facilitating integration into creative workflows. However, development has been hindered by the lack of large-scale public symbolic music datasets. The MetaScore dataset fills this gap by collecting musical scores and metadata from the MuseScore forum.
Core Problem
The core challenge in symbolic music generation is the lack of large-scale datasets supporting complex music attribute controls. Existing methods struggle to provide natural language interaction interfaces, limiting user experience.
Innovation
We introduce the MetaScore dataset, combined with LLM-generated pseudo-natural language descriptions, supporting text and tag-controlled music generation. A new music representation method enables multiple music attribute controls.
Methodology
- �� Use LLM to generate music descriptions
- �� Train text-controlled MetaScore Transformer-Text model
- �� Train tag-controlled MetaScore Transformer-Tags model
- �� Provide multiple music attribute controls, including instrument, style, composer, and complexity
Experiments
Experiments use the MetaScore dataset for training, comparing text and tag models against baseline models. Listening tests evaluate model performance in coherence, arrangement, adherence, and overall quality.
Results
Experimental results show that text and tag models outperform baseline models in listening tests. Text models excel in natural language interaction, while tag models perform well in coherence and arrangement.
Applications
These models can be used in music creation and education, offering natural language interaction interfaces, allowing users to generate music based on their needs.
Limitations & Outlook
Models struggle with complex free-text inputs, potentially degrading music quality. Auto-generated tags in the dataset may lack accuracy, affecting tag model performance.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. The MetaScore dataset is like a large pantry of ingredients, and LLM-generated descriptions are recipes. Our models are like chefs, selecting ingredients based on recipes to create music that suits your taste. The text model is like a chef who understands your preferences, while the tag model cooks based on preset ingredient choices.
ELI14 Explained like you're 14
Hey, imagine you're playing a game, MetaScore is like a super game library, and LLM-generated descriptions are game guides. Our models are like game characters, choosing weapons and skills based on guides to play the music you want. The text model is like a character who understands your thoughts, while the tag model fights based on preset weapons and skills.
Glossary
LLM (Large Language Model)
A machine learning model capable of understanding and generating natural language.
Used to generate natural language descriptions for music.
MetaScore
A dataset containing rich music metadata and pseudo-natural language descriptions.
Used to train music generation models.
Symbolic Music
Editable music format, allowing users to further create.
Format of generated music.
Text-to-Music Generation
Process of generating music based on natural language input.
Main function of the models.
Tag Control
Process of controlling music generation through preset tags.
Auxiliary function of the models.
Open Questions Unanswered questions from this research
- 1 How to improve model handling of complex free-text inputs?
- 2 How to enhance accuracy of auto-tag generation?
Applications
Immediate Applications
Music Creation
Musicians can use models to generate music matching specific styles and complexities.
Long-term Vision
Music Education
Students can learn music creation through natural language interaction.
Abstract
Recent years have seen many audio-domain text-to-music generation models that rely on large amounts of text-audio pairs for training. However, symbolic-domain controllable music generation has lagged behind partly due to the lack of a large-scale symbolic music dataset with extensive metadata and captions. In this work, we present MetaScore, a new dataset consisting of 963K musical scores paired with rich metadata, including free-form user-annotated tags, collected from an online music forum. To approach text-to-music generation, We employ a pretrained large language model (LLM) to generate pseudo-natural language captions for music from its metadata tags. With the LLM-enhanced MetaScore, we train a text-conditioned music generation model that learns to generate symbolic music from the pseudo captions, allowing control of instruments, genre, composer, complexity and other free-form music descriptors. In addition, we train a tag-conditioned system that supports a predefined set of tags available in MetaScore. Our experimental results show that both the proposed text-to-music and tags-to-music models outperform a baseline text-to-music model in a listening test. While a concurrent work Text2MIDI also supports free-form text input, our models achieve comparable performance. Moreover, the text-to-music system offers a more natural interface than the tags-to-music model, as it allows users to provide free-form natural language prompts.