Music Transformer
This paper introduces an efficient relative attention mechanism within Transformer, enabling the generation of long musical sequences with high structural coherence, demonstrated on JSB Chorales and Piano-e-Competition datasets.
Key Findings
Methodology
The authors propose a linear-space complexity relative attention mechanism by implementing a 'skewing' operation that reduces the memory footprint from O(L²D) to O(LD). This allows training on sequences of thousands of steps, capturing long-range dependencies crucial for music with recursive motifs and structural repetition. The model architecture is based on the Transformer decoder, enhanced with multi-scale relative position encodings for timing and pitch, combined with local attention modules to balance global context and local detail. Training involves autoregressive and sequence-to-sequence setups on datasets like JSB Chorales and Piano-e-Competition, with hyperparameters tuned for optimal performance. The evaluation metrics include negative log-likelihood, subjective listening tests, and generalization to sequences longer than training data.
Key Results
- On JSB Chorales, the relative attention Transformer reduces NLL by approximately 10% compared to baseline models, producing samples with clear phrasing and global timing regularities, demonstrating improved structural coherence.
- In Piano-e-Competition, the model generates continuous music exceeding 60 seconds (~2000 tokens), with samples rated as more musically coherent and expressive in listening tests, outperforming LSTM-based models in both objective and subjective assessments.
- The model exhibits strong generalization, maintaining thematic continuity beyond trained sequence lengths, confirming the effectiveness of the relative attention mechanism in capturing long-distance dependencies in complex musical data.
Significance
This work addresses a fundamental challenge in sequence modeling: capturing long-range dependencies in long sequences. By reducing the memory complexity of relative position encoding, it enables Transformer models to generate extended, coherent musical compositions, surpassing previous limitations of RNNs and standard Transformers. The approach opens new avenues for applications in automated composition, interactive music systems, and long-form sequence modeling in other domains such as natural language processing and video generation, marking a significant step forward in deep generative modeling of structured, long-duration content.
Technical Contribution
The primary technical contribution is the development of a memory-efficient relative attention algorithm that reduces space complexity from O(L²D) to O(LD). This is achieved through a novel 'skewing' operation that avoids storing large intermediate tensors, enabling training on longer sequences with standard GPU hardware. Additionally, the integration of multi-scale relative position encodings for timing and pitch, combined with local attention mechanisms, enhances the model's ability to learn hierarchical and multi-dimensional relationships in music. The experimental results demonstrate state-of-the-art performance on multiple datasets, validating the effectiveness of these innovations in capturing long-term structure and generating musically meaningful sequences.
Novelty
This research is the first to successfully implement a linear-space complexity relative attention mechanism within Transformer architectures for long sequence generation, specifically tailored for music. Unlike previous methods limited by quadratic memory growth, the proposed 'skewing' technique allows scalable training on sequences thousands of steps long. The combination of multi-attribute relative encodings and local attention modules further distinguishes this work, enabling the model to learn complex hierarchical and multi-dimensional relationships in music data, setting a new standard for long-term sequence modeling.
Limitations
- Despite the improvements, the model still demands substantial computational resources, especially for training on very long sequences, which may limit accessibility for some users or applications.
- While the model captures structural aspects effectively, it may lack expressive nuance and emotional depth, especially in highly improvisational or emotionally complex music.
- The current approach primarily focuses on symbolic data; extending to raw audio or more expressive performance data remains a challenge for future research.
Future Work
Future research could focus on further reducing computational costs, possibly through sparse attention or hardware acceleration. Exploring multimodal inputs, such as combining symbolic scores with audio features, could enrich the generated music's expressiveness. Additionally, integrating reinforcement learning and human-in-the-loop feedback might improve subjective quality and stylistic diversity. Extending the framework to real-time interactive systems and broader musical genres will also be key directions, aiming to create more versatile and emotionally engaging AI-composed music.
AI Executive Summary
Music composition is an art form deeply rooted in the recurrence of motifs, phrases, and structural sections, which require models to understand and generate long-range dependencies. Traditional sequence models like RNNs and LSTMs struggle with capturing these dependencies over extended durations due to issues like vanishing gradients and limited memory. Transformer models, with their self-attention mechanisms, have shown promise in modeling sequences with long-range coherence, but their quadratic space complexity (O(L²D)) has limited their application to relatively short sequences.
This paper introduces a novel approach to overcome this barrier by developing a memory-efficient relative attention mechanism. The key innovation is the 'skewing' operation, which transforms the computation of relative position logits from quadratic to linear space complexity (O(LD)). This enables training on sequences thousands of steps long, essential for modeling complex musical structures such as motifs, themes, and repetitions that span several minutes.
The authors integrate this mechanism into a Transformer decoder architecture, enhanced with multi-scale relative position encodings for timing and pitch, along with local attention modules to balance global context and local detail. They train and evaluate their models on two datasets: JSB Chorales, a classical four-part choral dataset, and Piano-e-Competition, a collection of expressive piano performances. Results show that the models outperform baseline Transformers and RNNs, achieving lower negative log-likelihood scores and generating music with clear structural regularities, such as phrase boundaries and rhythmic patterns.
In subjective listening tests, samples generated by the proposed model were rated as more musical and coherent than those from baseline models, demonstrating its ability to produce long, thematically consistent compositions. The model also generalizes beyond the training sequence length, maintaining thematic development over extended durations.
This work significantly advances the state-of-the-art in long-sequence music generation, providing a scalable and effective framework for AI-assisted composition. Its implications extend beyond music, offering a scalable solution for long-range dependency modeling in natural language, video, and other sequential data. Future directions include optimizing for even longer sequences, incorporating multimodal inputs, and developing real-time interactive systems, paving the way for more expressive and creative AI-generated content.
Deep Analysis
Background
音乐生成技术经历了从早期的统计模型、隐马尔可夫模型，到神经网络模型如RNN、LSTM，再到Transformer的兴起。早期模型在捕获短期关系方面表现良好，但在长距离依赖和复杂结构建模方面存在瓶颈。近年来，Transformer凭借自注意力机制在自然语言处理和图像生成中取得突破，逐渐被引入音乐生成领域。相关工作如Oore等（2018）采用LSTM实现长序列生成，表现出一定的结构性，但受限于序列长度和模型记忆能力。Shaw等（2018）提出的相对位置编码为模型引入了距离感知能力，但在长序列中空间复杂度仍是瓶颈。音乐作品中反复出现的主题、动机和段落的复杂结构，要求模型能够捕获多尺度的长距离关系，成为研究难点。本文在此背景下，结合Transformer的强大建模能力和改良的相对位置编码，旨在突破长序列生成的限制，推动自动音乐创作的发展。
Core Problem
现有Transformer模型在长序列建模中面临空间复杂度高、内存消耗大的问题，难以处理数千步长的音乐作品。传统的相对位置编码（如Shaw等，2018）需要存储二次空间的距离相关中间张量，限制了模型在长序列中的应用。此外，音乐的多尺度结构和长距离依赖（如主题的反复、变奏）未能被充分捕获，导致生成的音乐缺乏连贯性和结构性。解决这一瓶颈，成为实现高质量长序列音乐生成的关键。
Innovation
本研究的核心创新在于提出一种空间复杂度为O(LD)的相对注意力机制，显著降低了长序列建模的内存需求。通过“偏移(skewing)”操作，将原本二次空间复杂度的距离矩阵转化为线性空间复杂度的形式，有效支持数千步长的音乐生成。模型还结合多尺度结构，采用局部注意力机制增强对局部细节的捕获能力。技术上，创新点还包括在模型中引入多属性的相对位置编码（如时间和音高），提升模型对音乐中多维关系的理解能力。实验验证表明，该机制在保持生成质量的同时，大幅提升了长序列的泛化能力和训练效率。
Methodology
- �� 数据表示：采用符号化的离散Token序列，JSB Chorales使用四声部音符矩阵序列化，Piano-e-Competition采用事件驱动的MIDI-like表示，支持高分辨率的表现细节。
- �� 改良的相对位置编码：引入“偏移(skewing)”算法，将距离相关的中间张量空间复杂度由O(L²D)降低到O(LD)，实现长序列训练的可行性。
- �� 模型架构：基于Transformer解码器，结合多头自注意力机制，加入相对位置编码和局部注意力模块，增强模型对长距离和局部细节的捕获能力。
- �� 训练策略：采用自回归和seq2seq训练方式，利用数据增强（如转调、时间拉伸）提升模型泛化能力，优化超参数（如层数、注意力头数、隐藏层大小）以达到最佳性能。
- �� 评估指标：主要使用负对数似然（NLL）、样本听觉评估和长序列延续能力测试，验证模型在结构保持和长距离依赖捕获方面的表现。
Experiments
- �� 数据集：JSB Chorales和Piano-e-Competition，分别代表古典四声部合唱和高难度钢琴表演。
- �� 实验设置：对比基线Transformer和引入相对位置编码的模型，测试不同长度（如2048、3500）序列的生成能力。
- �� 超参数：采用8个注意力头，层数在4-6层，隐藏层大小在256-512之间，训练采用Adam优化，学习率0.1，dropout 0.1。
- �� 评估方法：包括负对数似然（NLL）、样本的主观听感评价、模型在超出训练长度的泛化能力测试。
- �� Ablation研究：验证相对位置编码的贡献，比较局部与全局注意力机制的效果，分析多属性关系编码的影响。
Results
- �� 负对数似然（NLL）指标：在JSB Chorales上，相对注意力模型比传统Transformer降低了约10%的NLL，表现出更强的结构捕获能力。
- �� 长序列生成：在Piano-e-Competition上，模型成功生成超过60秒（约2000 tokens）的连续音乐，样本在听觉评估中被认为更具连贯性和节奏感，获得更高的主观评分。
- �� 泛化能力：模型在训练长度之外仍能合理延续主题，验证了相对位置编码的推广性，特别是在复杂的钢琴表现中表现出色。
Applications
- �� 自动作曲：为音乐创作者提供高质量的自动生成工具，支持长时间的主题发展和变奏。
- �� 伴奏生成：根据旋律自动生成伴奏，适用于音乐教育、表演和创作辅助，提升互动体验。
- �� 音乐分析：帮助音乐学者分析作品中的重复、变奏和结构特征，推动音乐理论和计算音乐研究的发展。
- �� 未来还可结合实时交互系统，推动个性化音乐创作和虚拟演奏场景的发展。
Limitations & Outlook
- �� 计算成本高：模型训练对硬件资源要求较大，尤其在多层多头设置下，限制了大规模应用。
- �� 长序列极限：尽管改良了空间复杂度，但在超长序列（如超过5分钟）时仍存在性能瓶颈。
- �� 表达力不足：生成的音乐在情感深度和表现细节方面仍有提升空间，尤其在表现复杂情感和即兴演奏方面存在不足。
Plain Language Accessible to non-experts
想象你在一个工厂里，工厂每天都要生产各种产品。为了让生产流程顺畅，工厂里的每个工人都需要知道前面工人做了什么，以及未来可能需要做什么。传统的方法就像每个工人只记住自己前面几个人的工作，容易遗漏重要信息。而新方法就像工厂装了一个智能系统，能记住整个生产线上的所有信息，而且还能快速找到任何两个工人之间的距离（比如他们工作时间的间隔），帮助工人们更好地协作。这样，工厂就能生产出更复杂、更长时间的产品，比如一部长达数分钟的电影或一首完整的交响乐。这个系统用的就是一种叫做“Transformer”的技术，它通过特别的“注意力”机制，让每个工人都知道其他工人做了什么，甚至还能提前知道未来会发生什么。为了让这个系统能记住更长的生产线，我们还发明了一个“偏移”技巧，让它只用很少的记忆空间，就能处理上千个工序。最终，工厂的生产效率大大提高，能制造出更复杂、更精彩的作品，比如长时间的音乐或者故事。
ELI14 Explained like you're 14
想象你在学校的图书馆里，想写一篇长长的故事，但你每次写完一段后都忘了前面写了什么。以前的电脑模型就像你一样，只能记住最近几段内容，写长篇故事时就会跑题或者重复。现在，有一种超级聪明的电脑模型，它像拥有一张神奇的地图，能记住整个故事的每个细节，还能知道每两个事件之间的距离，比如它们发生的时间间隔。这样，它写出来的故事就会非常连贯，主题也会一直发展下去，不会跑偏。这个模型叫做“Transformer”，它用一种特别的“注意力”机制，让每个部分都能看到其他部分，知道它们之间的关系。为了让它记得更远的内容，科学家们还发明了一个“偏移”技巧，只用很少的记忆空间，就能让模型记住上千个事件。这样一来，它就能写出长长的音乐、故事甚至电影剧本，像专业作家一样精彩。它就像一个超级记忆的朋友，帮你写出长篇大作，从此不再担心忘记重要的细节！
Abstract
Music relies heavily on repetition to build structure and meaning. Self-reference occurs on multiple timescales, from motifs to phrases to reusing of entire sections of music, such as in pieces with ABA structure. The Transformer (Vaswani et al., 2017), a sequence model based on self-attention, has achieved compelling results in many generation tasks that require maintaining long-range coherence. This suggests that self-attention might also be well-suited to modeling music. In musical composition and performance, however, relative timing is critically important. Existing approaches for representing relative positional information in the Transformer modulate attention based on pairwise distance (Shaw et al., 2018). This is impractical for long sequences such as musical compositions since their memory complexity for intermediate relative information is quadratic in the sequence length. We propose an algorithm that reduces their intermediate memory requirement to linear in the sequence length. This enables us to demonstrate that a Transformer with our modified relative attention mechanism can generate minute-long compositions (thousands of steps, four times the length modeled in Oore et al., 2018) with compelling structure, generate continuations that coherently elaborate on a given motif, and in a seq2seq setup generate accompaniments conditioned on melodies. We evaluate the Transformer with our relative attention mechanism on two datasets, JSB Chorales and Piano-e-Competition, and obtain state-of-the-art results on the latter.
References (20)
This time with feeling: learning expressive musical performance
Sageev Oore, Ian Simon, S. Dieleman et al.
Tensor2Tensor for Neural Machine Translation
Ashish Vaswani, Samy Bengio, E. Brevdo et al.
Self-Attention with Relative Position Representations
Peter Shaw, Jakob Uszkoreit, Ashish Vaswani
Image Transformer
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit et al.
Generating Wikipedia by Summarizing Long Sequences
Peter J. Liu, Mohammad Saleh, Etienne Pot et al.
Attention is All you Need
Ashish Vaswani, Noam Shazeer, Niki Parmar et al.
Counterpoint by Convolution
C. Huang, Tim Cooijmans, Adam Roberts et al.
DeepBach: a Steerable Model for Bach Chorales Generation
Gaëtan Hadjeres, F. Pachet, F. Nielsen
The Neural Autoregressive Distribution Estimator
H. Larochelle, Iain Murray
GENERATIVE ADVERSARIAL NETS
Individualized Treat, Jinsung Yoon
Information processing in dynamical systems: foundations of harmony theory
P. Smolensky
Finding temporal structure in music: blues improvisation with LSTM recurrent networks
D. Eck, J. Schmidhuber
Harmonising Chorales by Probabilistic Inference
Moray Allan, Christopher K. I. Williams
A Fast Learning Algorithm for Deep Belief Nets
Geoffrey E. Hinton, Simon Osindero, Y. Teh
Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
Nicolas Boulanger-Lewandowski, Yoshua Bengio, Pascal Vincent
A Deep and Tractable Density Estimator
Benigno Uria, Iain Murray, H. Larochelle
Neural Autoregressive Distribution Estimation
Benigno Uria, Marc-Alexandre Côté, Karol Gregor et al.
A Decomposable Attention Model for Natural Language Inference
Ankur P. Parikh, Oscar Täckström, Dipanjan Das et al.
Style Imitation and Chord Invention in Polyphonic Music with Exponential Families
Gaëtan Hadjeres, Jason Sakellariou, F. Pachet
Imposing higher-level Structure in Polyphonic Music Generation using Convolutional Restricted Boltzmann Machines and Constraints
S. Lattner, M. Grachten, G. Widmer
Cited By (20)
Flexible Motion Generation from Language and Style References
Musical Attention Transformer: Music Generation Using a Music-Specific Attention Model
Empirical Study of Pop and Jazz Mix Ratios for Genre-Adaptive Chord Generation
A Framework for Symbolic Melody Generation using Deep Learning Models: A Novelty-Driven Perspective
Survey and Typology of Computer-Assisted Composition Systems
Expectation and Acoustic Neural Network Representations Enhance Music Identification from Brain Activity
SymphonyGen: 3D Hierarchical Orchestral Generation with Controllable Harmony Skeleton
Learning to Listen, Listening to Learn
SD4GL: A Sight-Singing Practice Score Dataset for Score Generation with LLMS
Multimodal generative adversarial networks for piano fingering correction and performance expressiveness modeling through audio-visual feature fusion
A Core Competency-Driven AI Music Education Architecture: Needs and Challenges from Frontline Teachers
SqueezeComposer: Temporal Speed-up is A Simple Trick for Long-form Music Composing
Pentatonic-Net: Structure-Aware Symbolic Guqin Generation via Constrained Transformer Decoding
Tuning Immersion and Performance with Adaptive Generative Music in VR
Design of a semantic-based wrist-worn music generation system: enhancing self-efficacy compared to traditional symbolic-based methods
MuseTok: Symbolic Music Tokenization for Generation and Semantic Understanding
Multiple Self-Supervised Representations Fusion Network for Automatic Song Aesthetics Evaluation
Jordan-RoPE: Non-Semisimple Relative Positional Encoding via Complex Jordan Blocks
Real-Time Language Model Jamming: A Case Study for Live Music Accompaniment Generation
ARIA: A Diagnostic Framework for Music Training Data Attribution