Impact of time and note duration tokenizations on deep learning symbolic music modeling
This study compares time and note duration tokenizations, showing explicit info improves model performance across tasks.
Key Findings
Methodology
Using a Transformer architecture, four tokenization schemes (TS+Dur, TS+NOff, Pos+Dur, Pos+NOff) were compared across music generation, classification, and sequence embedding tasks. The study adjusted how time and duration are represented, evaluating their impact on prediction accuracy, embedding quality, and generation coherence. Experiments utilized datasets like GiantMIDI, POP909, and EMOPIA, measuring prediction error rates, classification accuracy, and embedding isotropy. The analysis revealed that explicit time shift combined with note duration (TS+Dur) yielded the lowest error rates (as low as 0.001) and highest classification accuracy (~97.3%), while position-based schemes excelled in certain embedding tasks.
Key Results
- Explicit time shift plus note duration (TS+Dur) achieved the best prediction accuracy with error ratios below 0.002, outperforming other schemes in autoregressive generation. Classification accuracy was approximately 97.3%, indicating robust performance.
- Position-based tokenizations, while reducing type errors, caused irregular note onset distributions, leading to overlapping notes and less rhythmic consistency in generated music.
- Sequence embedding experiments showed Pos+Dur tokens produced more evenly distributed embeddings with higher intrinsic dimensions (~177), suggesting better capture of global musical structure.
Significance
This research highlights how explicit encoding of temporal and duration information significantly influences deep learning models' ability to understand and generate music. It provides practical insights for designing more effective tokenization schemes tailored to specific tasks, advancing both theoretical understanding and practical applications in symbolic music AI. The findings support the development of more accurate, expressive, and controllable music generation systems, addressing longstanding challenges in music modeling.
Technical Contribution
The paper systematically compares four tokenization strategies, integrating specific algorithms like Byte Pair Encoding for vocabulary optimization. It introduces a novel error analysis metric (TSE) to quantify prediction reliability, and demonstrates that combining explicit time shift and note duration enhances model robustness. The work also provides empirical evidence linking tokenization choices to embedding space properties, offering new perspectives on sequence representation learning in music AI.
Novelty
This is the first comprehensive empirical comparison of time and note duration tokenizations across multiple tasks, establishing the superiority of explicit temporal encoding in generation and classification. The integration of prediction error metrics and embedding isotropy analysis offers new insights into tokenization impacts, filling a gap in existing literature that often lacks such systematic evaluation.
Limitations
- The experiments focus primarily on monophonic, piano-like datasets, limiting generalization to polyphonic, multi-instrument, or real-world audio scenarios. Future work should explore multi-track and multi-modal extensions.
- High computational costs associated with training large Transformer models restrict scalability and real-time application potential. Optimization for efficiency remains necessary.
- While explicit time and duration tokens improve certain tasks, they may reduce flexibility in expressive performance or improvisation contexts, requiring further adaptation for broader musical styles.
Future Work
Future research will extend tokenization schemes to multi-track, multi-modal, and real-time contexts, integrating semantic and expressive features. Developing adaptive tokenization methods that dynamically balance explicit and implicit representations could further enhance model versatility. Additionally, applying these strategies to music transcription, synthesis, and interactive systems will validate their broader utility and robustness.
AI Executive Summary
This work addresses a fundamental challenge in symbolic music modeling: how to effectively tokenize music for deep learning. As Transformer-based models dominate tasks like music generation, transcription, and classification, the choice of tokenization strategy becomes crucial. Traditional approaches often rely on implicit or simplistic representations of time and note durations, which limit the model's ability to capture rhythmic and structural nuances.
The authors systematically compare four tokenization schemes combining two key dimensions: time representation (TimeShift vs. Position) and note duration (Duration vs. NoteOff). They evaluate these schemes across multiple tasks—music generation on the POP909 dataset, composer and emotion classification on GiantMIDI and EMOPIA, and sequence embedding learning—using metrics like prediction error ratios, classification accuracy, and embedding isotropy. Results show that explicit time shift combined with note duration (TS+Dur) consistently outperforms other schemes, achieving the lowest prediction errors (as low as 0.001) and high classification accuracy (~97.3%). Conversely, position-based schemes excel in embedding uniformity but struggle with rhythmic consistency.
The analysis reveals that explicit encoding of temporal and duration information enhances the model’s understanding of musical structure, rhythm, and melody. In generation tasks, explicit time tokens produce more accurate and rhythmically coherent outputs, while in classification, explicit durations help models distinguish complex harmonic patterns. Embedding analysis indicates that schemes with explicit position and duration produce more evenly distributed representations, facilitating better generalization.
Overall, this research underscores the importance of explicit temporal and duration encoding in symbolic music AI. It provides practical guidelines for designing tokenization schemes tailored to specific tasks, paving the way for more expressive and reliable music generation and analysis systems. Future work will explore multi-instrument, multi-modal, and real-time applications, aiming to further improve the robustness and versatility of symbolic music models.
Deep Dive
Abstract
Symbolic music is widely used in various deep learning tasks, including generation, transcription, synthesis, and Music Information Retrieval (MIR). It is mostly employed with discrete models like Transformers, which require music to be tokenized, i.e., formatted into sequences of distinct elements called tokens. Tokenization can be performed in different ways. As Transformer can struggle at reasoning, but capture more easily explicit information, it is important to study how the way the information is represented for such model impact their performances. In this work, we analyze the common tokenization methods and experiment with time and note duration representations. We compare the performances of these two impactful criteria on several tasks, including composer and emotion classification, music generation, and sequence representation learning. We demonstrate that explicit information leads to better results depending on the task.
References (20)
Byte Pair Encoding for Symbolic Music
Nathan Fradet, Jean-Pierre Briot, Fabien Chhel et al.
Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions
Yu-Siang Huang, Yi-Hsuan Yang
GiantMIDI-Piano: A large-scale MIDI dataset for classical piano music
Qiuqiang Kong, Bochen Li, Jitong Chen et al.
LakhNES: Improving Multi-instrumental Music Generation with Cross-domain Pre-training
Chris Donahue, H. H. Mao, Yiting Li et al.
On the evaluation of generative models in music
Li-Chia Yang, Alexander Lerch
POP909: A Pop-Song Dataset for Music Arrangement Generation
Ziyu Wang, K. Chen, Junyan Jiang et al.
Attention is All you Need
Ashish Vaswani, Noam Shazeer, Niki Parmar et al.
An Algorithm for Finding Intrinsic Dimensionality of Data
K. Fukunaga, D. R. Olsen
FIGARO: Controllable Music Generation using Learned and Expert Features
Dimitri von Rütte, Luca Biggio, Yannic Kilcher et al.
Reasoning with Transformer-based Models: Deep Learning, but Shallow Reasoning
Chadi Helwe, C. Clavel, Fabian M. Suchanek
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee et al.
Folk music style modelling by recurrent neural networks with long short term memory units
Bob L. Sturm, J. F. Santos, I. Korshunova
Adam: A Method for Stochastic Optimization
Diederik P. Kingma, Jimmy Ba
This time with feeling: learning expressive musical performance
Sageev Oore, Ian Simon, S. Dieleman et al.
Mixed Precision Training
P. Micikevicius, Sharan Narang, Jonah Alben et al.
Estimating the effective dimension of large biological datasets using Fisher separability analysis
Luca Albergante, Jonathan Bac, A. Zinovyev
The Curious Case of Neural Text Degeneration
Ari Holtzman, Jan Buys, Li Du et al.
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Nils Reimers, Iryna Gurevych
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
O. Khattab, M. Zaharia
Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
Tongzhou Wang, Phillip Isola
Cited By (11)
How Far Should Tokenization Go? Predictive Effectiveness and Relational Losslessness
Inspecting Deep Learning Techniques for Music Transcription
On the de-duplication of the Lakh MIDI dataset
Moonbeam: A MIDI Foundation Model Using Both Absolute and Relative Music Attributes
Evaluating Interval-based Tokenization for Pitch Representation in Symbolic Music Analysis
MidiTok Visualizer: a tool for visualization and analysis of tokenized MIDI symbolic music
Do we need more complex representations for structure? A comparison of note duration representation for Music Transformers
Automated Thematic Composer Classification Using Segment Retrieval
Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: A Survey
Quantifying Repetition in Symbolic Music using Lempel-Ziv Compression
Byte Pair Encoding for Symbolic Music