Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation
Museformer combines fine- and coarse-grained attention, enabling modeling of long music sequences with improved structural coherence, surpassing 3× longer sequences.
Key Findings
Methodology
The approach integrates structure-aware Bars selected via similarity statistics with multi-head attention. Fine-grained attention directly attends to key Bars, capturing long-distance relations, while other Bars are summarized into tokens for coarse attention, reducing complexity. The model employs multiple Transformer layers with positional embeddings, utilizing a dynamic selection of structure-related Bars based on statistical similarity (e.g., Jaccard index). The training maximizes likelihood of sequential token prediction, optimizing for both musical continuity and structural fidelity.
Key Results
- On sequences of 10,240 tokens, Museformer achieves a perplexity (PPL) of 1.35, outperforming Transformer-XL (1.43) and Linear Transformer (1.64), demonstrating superior long-sequence modeling capacity.
- Structural similarity error (SE) drops to 0.95%, significantly better than Longformer (5.25%), indicating more realistic structural reproduction.
- Subjective listening tests show Museformer scores highest across musicality, short-term and long-term structure, with a preference rate of 46%, statistically significant (p<0.05).
Significance
This work addresses the core challenge of long-range structure modeling in music generation, overcoming the quadratic complexity of full attention. By selectively focusing on structure-relevant Bars and summarizing others, it enables high-quality, long-duration music synthesis. The approach advances AI's ability to generate coherent, artistically rich compositions, with implications extending to NLP and other long-sequence tasks, fostering broader AI creativity.
Technical Contribution
The key technical innovation lies in the similarity-based selection of structure-relevant Bars, combined with a multi-scale attention mechanism that balances detail and efficiency. The model's architecture integrates summarization tokens and dynamic attention masks, providing theoretical guarantees for long sequence modeling while maintaining computational feasibility. This design opens new avenues for scalable, structure-aware sequence generation.
Novelty
This is the first to leverage statistical similarity measures for dynamic selection of structure-related Bars in music, integrating them into a multi-scale attention framework. Unlike prior models that rely solely on fixed or content-based attention, this approach explicitly encodes long-distance repetitive patterns, significantly improving structural coherence and long-term dependencies in generated music.
Limitations
- The structure-related Bars are selected based on static statistics, which may not adapt well to highly experimental or non-repetitive music styles, potentially limiting diversity.
- Handling sequences beyond 30,000 tokens still poses computational challenges, requiring further hardware optimization.
- Cross-genre generalization and adaptation to non-Western musical traditions need further validation, as the current similarity metrics are tuned for Western tonal music.
Future Work
Future research will explore adaptive, context-aware selection of structure Bars, possibly incorporating reinforcement learning. Integrating multimodal cues like rhythm and lyrics could further enrich the generated music. Additionally, extending the framework to other sequence domains, such as long-form storytelling or dialogue, offers promising directions.
AI Executive Summary
Music composition has long been a complex challenge for artificial intelligence, especially when it involves creating long, coherent pieces with intricate structures. Traditional Transformer models, while powerful, struggle with sequences exceeding thousands of tokens due to their quadratic attention complexity. This limitation hampers their ability to model essential long-range dependencies like repetitions and variations, which are fundamental to musical form.
To address this, the paper introduces Museformer, a novel Transformer architecture that employs a dual attention mechanism—fine-grained attention for structure-related Bars and coarse-grained attention for others. The structure-related Bars are selected based on similarity statistics derived from human-composed music, capturing long-distance repetitions and thematic links. These Bars are directly attended to, enabling the model to learn and reproduce complex musical structures. Meanwhile, less critical Bars are summarized into tokens, significantly reducing computational load.
Experimental results on the Lakh MIDI dataset demonstrate that Museformer can handle sequences over three times longer than previous models, with perplexity dropping to 1.35 at 10,240 tokens and structure similarity errors halved compared to baselines. Subjective evaluations further confirm its superior musicality and structural coherence, with the highest preference scores among tested models. This breakthrough not only enhances AI's ability to generate realistic, long-form music but also provides a scalable framework for long-sequence modeling in other domains.
Looking ahead, the authors plan to develop adaptive, context-sensitive structure selection methods, incorporate multimodal cues, and extend the approach to broader AI creative tasks. Despite current limitations in genre adaptability and extreme sequence lengths, Museformer marks a significant step toward more intelligent, artistically expressive AI-generated music, promising a future where machines compose with human-like understanding of musical form and emotion.
Deep Analysis
Background
符号音乐生成经历了从规则基础到深度学习的演变，Transformer的引入极大提升了建模能力。Huang等的Music Transformer首次展示了Transformer在音乐中的潜力，但其全注意力机制在长序列中存在二次复杂度，限制了其扩展性。后续研究如Transformer-XL和Longformer尝试缓解这一问题，但在捕获音乐中的长距离结构关系方面仍有不足，尤其是在重复、变奏等结构特征的建模上。随着音乐序列的增长，模型需要同时处理内容的连续性和结构的复杂性，提出更高效且结构敏感的模型成为亟需解决的问题。
Core Problem
核心挑战在于长序列音乐的结构建模与高效处理。全注意力机制在序列长度增加时计算成本急剧上升，难以应用于实际长序列音乐生成。同时，音乐具有丰富的结构特征，如重复、变奏和远距离联系，传统模型难以捕获这些长距离关系，导致生成的音乐缺乏合理的结构连贯性。这两个问题的结合，限制了自动作曲技术的实际应用和艺术表现力。
Innovation
本文的创新主要在于引入基于音乐统计的结构相关Bars选择机制，结合细粒度与粗粒度注意力，形成多尺度长序列建模框架。具体包括：• 结构相关Bars的统计选择，利用相似性指标筛选关键Bars，增强结构捕获能力；• 细粒度注意力，直接关注结构Bars，学习长距离关系；• 粗粒度注意力，通过总结Token压缩其他Bars信息，降低复杂度；• 多层Transformer架构，结合总结Token实现高效长序列建模。这些创新突破了传统全注意力的限制，兼顾结构与内容，显著提升生成质量。
Methodology
- �� 输入：音乐Token序列X，按Bar划分，每个Bar包含多个Token。• 结构Bars选择：通过计算相似性（如Jaccard）筛选出与当前Bar关系密切的关键Bars（如前1、2、4、8、12、16、24、32个Bars）。• 结构相关Bars的细粒度注意力：每个Token直接关注这些Bars，学习长距离结构关系。• 其他Bars用总结Token表示，进行粗粒度注意力，压缩信息。• 通过在每个Bar后插入总结Token，形成X1, s1, ..., Xb, sb序列。• 多层Transformer处理，结合位置编码，输出下一Token预测。• 训练目标最大化生成音乐的连续性和结构合理性，采用交叉熵损失。• 采用多头注意力机制，结合相似性统计进行结构Bars的动态选择。
Experiments
- �� 数据集：Lakh MIDI，包含近3万首多乐器音乐，平均每首95Bar。• 表示：采用REMI-like Token化，包括音高、时值、乐器等信息。• 训练：采用Adam优化器，学习率逐步升高后下降，批次4首。• 评估：困惑度（PPL）和结构相似性误差（SE），在不同序列长度（1024、5120、10240）上比较。• 对比模型：Music Transformer、Transformer-XL、Longformer、Linear Transformer。• 还进行主观听感评估，邀请专业人士打分。
Results
- �� 在长序列（10,240 Token）上，Museformer PPL降至1.35，优于对比模型（如Transformer-XL的1.43），表现出优异的长序列建模能力。• 结构相似性误差（SE）为0.95%，远低于Longformer的5.25%，说明生成音乐结构更贴近人类作品。• 主观评估中，Museformer在音乐性、短期与长期结构方面得分最高，偏好率达46%，统计学显著（p<0.05）。• Ablation研究显示，结构Bars选择和总结Token机制对性能提升至关重要。
Applications
该模型适用于自动作曲、音乐伴奏生成、音乐教育等场景。只需提供基础Token序列，即可生成具有复杂结构的长篇音乐作品。未来可结合多模态信息（如歌词、节奏）实现更丰富的音乐创作。其高效长序列建模能力，有望推动音乐AI在商业、娱乐、创作辅助等方面的应用，提升自动作曲的艺术表现力。
Limitations & Outlook
模型依赖统计特征进行结构Bars选择，可能在非典型或复杂音乐中表现不足。长序列（超3万Token）时仍面临硬件限制。对不同风格音乐的泛化能力有限，未来需引入自适应机制和多模态信息以增强鲁棒性。
Abstract
Symbolic music generation aims to generate music scores automatically. A recent trend is to use Transformer or its variants in music generation, which is, however, suboptimal, because the full attention cannot efficiently model the typically long music sequences (e.g., over 10,000 tokens), and the existing models have shortcomings in generating musical repetition structures. In this paper, we propose Museformer, a Transformer with a novel fine- and coarse-grained attention for music generation. Specifically, with the fine-grained attention, a token of a specific bar directly attends to all the tokens of the bars that are most relevant to music structures (e.g., the previous 1st, 2nd, 4th and 8th bars, selected via similarity statistics); with the coarse-grained attention, a token only attends to the summarization of the other bars rather than each token of them so as to reduce the computational cost. The advantages are two-fold. First, it can capture both music structure-related correlations via the fine-grained attention, and other contextual information via the coarse-grained attention. Second, it is efficient and can model over 3X longer music sequences compared to its full-attention counterpart. Both objective and subjective experimental results demonstrate its ability to generate long music sequences with high quality and better structures.
References (20)
Longformer: The Long-Document Transformer
Iz Beltagy, Matthew E. Peters, Arman Cohan
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas et al.
Attention is All you Need
Ashish Vaswani, Noam Shazeer, Niki Parmar et al.
Music Transformer: Generating Music with Long-Term Structure
C. Huang, Ashish Vaswani, Jakob Uszkoreit et al.
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context
Zihang Dai, Zhilin Yang, Yiming Yang et al.
Generating Long Sequences with Sparse Transformers
R. Child, Scott Gray, Alec Radford et al.
Image Transformer
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit et al.
Classical Music Composition Using State Space Models
Anna K. Yanchenko, Sayan Mukherjee
Improving Language Understanding by Generative Pre-Training
Alec Radford, Karthik Narasimhan
Learning-Based Methods for Comparing Sequences, with Applications to Audio-to-MIDI Alignment and Matching
Colin Raffel
Harmonising Chorales in the Style of Johann Sebastian Bach
Moray Allan
Analysis and Synthesis of Palestrina-Style Counterpoint Using Markov Chains
Mary Farbood, Bernd Schöner
SparTA: Deep-Learning Model Sparsity via Tensor-with-Sparsity-Attribute
Ningxin Zheng, Bin Lin, Quanlu Zhang et al.
LakhNES: Improving Multi-instrumental Music Generation with Cross-domain Pre-training
Chris Donahue, H. H. Mao, Yiting Li et al.
BP-Transformer: Modelling Long-Range Context via Binary Partitioning
Zihao Ye, Qipeng Guo, Quan Gan et al.
Compressive Transformers for Long-Range Sequence Modelling
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar et al.
Reformer: The Efficient Transformer
Nikita Kitaev, Lukasz Kaiser, Anselm Levskaya
Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions
Yu-Siang Huang, Yi-Hsuan Yang
SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive Connection
Xiaoya Li, Yuxian Meng, Qinghong Han et al.
Cited By (20)
A Survey on Evaluation Metrics for Music Generation
RPPNet: Perceptually-Grouped Rhythm-Pitch Primitives for Long-Term Structure Melody Generation via Boundary-Aware Modeling
Video background music generation using hybrid shared mixture-of-experts multimodal Transformer
Pengembangan Sistem Generator MIDI Berbasis Web Menggunakan Arsitektur RWKV
On the de-duplication of the Lakh MIDI dataset
A Novel Compressive Compound Word Encoding and Independent Word Attention for Symbolic Music Generation
Hamisfera: Sistem Rekomendasi Progresi Chord Berbasis Sentimen Lirik Melalui Studi Komparatif Arsitektur Transformer dan Mixture of Experts
Generating Symbolic Music from Natural Language Prompts using an LLM-Enhanced Dataset
Application and Research of Music Generation System Based on CVAE and Transformer-XL in Video Background Music
Evaluation of Pretrained Language Models on Music Understanding
Using Deep Learning for Text to Asia Music Generation
MMT-BERT: Chord-aware Symbolic Music Generation Based on Multitrack Music Transformer and MusicBERT
AE-AMT: Attribute-Enhanced Affective Music Generation With Compound Word Representation
A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives
Text2midi-InferAlign: Improving Symbolic Music Generation with Inference-Time Alignment
Artificial intelligence-based system for music generation
Aircraft Trajectory Dataset Augmentation in Latent Space
Adaptable Symbolic Music Infilling with MIDI-RWKV
Music generation in virtual reality based on grouping-combining algorithm model and multi-style chord music generation network
Multitrack Music Generation Combining Transformer and Diffusion Model