HeartMuLa: A Family of Open Sourced Music Foundation Models
HeartMuLa integrates four modules to achieve controllable music generation with 7B parameters, excelling in fidelity and structure.
Key Findings
Methodology
HeartMuLa consists of HeartCLAP (audio-text alignment), HeartTranscriptor (lyric recognition), HeartCodec (low-frame-rate music tokenizer), and HeartMuLa (multi-condition music generator). HeartCodec uses 12.5Hz compression with multi-layer RVQ for high-fidelity reconstruction.
Key Results
- HeartCodec achieves VISQOL score of 3.72, outperforming other codecs, with FAD as low as 0.27.
- HeartMuLa scores 4.62 in structural coherence at 7B parameters, surpassing SOTA models.
- ReFlow distillation improves inference efficiency to 10 steps while maintaining quality.
Significance
This research reproduces commercial-grade music generation systems using academic-scale data and GPUs, addressing semantic alignment and long-sequence modeling challenges in music generation.
Technical Contribution
Introduces HeartCodec, a low-frame-rate high-fidelity music tokenizer, enabling up to 6-minute song generation with fine-grained style control via natural language.
Novelty
HeartMuLa is the first to achieve end-to-end music generation combining text descriptions, lyrics, and reference audio, overcoming limitations in structural coherence and semantic alignment.
Limitations
- Limited capability in generating highly complex musical styles, requiring more annotated data.
- Inference efficiency, while improved, still needs optimization for real-time applications.
Future Work
Future work may explore more efficient compression algorithms, richer multimodal inputs, and extensions to real-time music generation and interactive tools.
AI Executive Summary
HeartMuLa is a family of open-source music foundation models addressing challenges in music generation and understanding. Existing systems struggle with semantic alignment, long-sequence modeling, and fine-grained control. HeartMuLa integrates four core modules—HeartCLAP, HeartTranscriptor, HeartCodec, and HeartMuLa—to provide comprehensive solutions.
HeartCodec employs 12.5Hz low-frame-rate compression with multi-layer RVQ for high-fidelity reconstruction, significantly enhancing generation quality. HeartMuLa supports up to 6-minute music generation and offers fine-grained style control, allowing users to specify styles for different sections using natural language. Experimental results show HeartMuLa outperforms existing models in metrics like VISQOL and FAD.
This research reproduces commercial-grade systems using academic-scale resources, providing a robust open-source baseline for multimodal content production and paving the way for future advancements in music intelligence.
Deep Analysis
Background
The field of music generation has seen significant progress, with notable works like Copet et al.'s multimodal models and Xu et al.'s audio representation learning. However, existing systems face challenges in long-sequence modeling and semantic alignment, limiting high-quality end-to-end music generation.
Core Problem
Current music generation models struggle to maintain structural coherence in long sequences and exhibit weak semantic alignment with text descriptions and audio references. Proprietary datasets further hinder reproducibility.
Innovation
HeartMuLa innovations include: 1) HeartCodec's 12.5Hz low-frame-rate compression for high-fidelity reconstruction; 2) fine-grained style control enabling natural language specification for different song sections; 3) end-to-end music generation combining text, lyrics, and reference audio.
Methodology
- �� HeartCLAP: learns shared embedding space for audio-text alignment.
- �� HeartTranscriptor: robust lyric recognition optimized for complex music signals.
- �� HeartCodec: employs multi-layer RVQ and ReFlow distillation for efficient high-fidelity music tokenization.
- �� HeartMuLa: generates long-form music with multi-condition inputs, supporting fine-grained style control.
Experiments
Experiments trained on 600,000 songs evaluated metrics like VISQOL and FAD against baselines. ReFlow distillation reduced inference steps to 10 while maintaining high-quality generation.
Results
HeartCodec achieves VISQOL score of 3.72, FAD of 0.27, and excels in reconstruction quality. HeartMuLa scores 4.62 in structural coherence at 7B parameters, outperforming SOTA models.
Applications
HeartMuLa is applicable to short video background music generation and music creation tools, especially for scenarios requiring fine-grained style control.
Limitations & Outlook
The model has limited capability in generating highly complex musical styles and requires optimization for real-time applications. Future work may explore efficient compression and real-time generation techniques.
Plain Language Accessible to non-experts
Imagine a music factory: HeartCLAP is the quality inspector ensuring lyrics match audio; HeartTranscriptor is the translator accurately recording lyrics; HeartCodec is the compressor shrinking music into small data blocks while preserving quality; HeartMuLa is the artist creating complete songs based on user descriptions, lyrics, and reference audio.
ELI14 Explained like you're 14
Imagine playing a music game where you choose lyrics, style, and even reference music, and the game generates a song just for you! HeartMuLa is like the super AI in the game—it understands your choices and creates high-quality music up to 6 minutes long. Cool, right?
Glossary
HeartCLAP
An audio-text alignment model learning shared embedding space for tagging and cross-modal retrieval.
Used for music semantic alignment and generation tasks.
HeartCodec
A music tokenizer using low-frame-rate compression for high-fidelity reconstruction.
Core component in the generation stage.
ReFlow Distillation
An optimization technique reducing inference steps from 50 to 10.
Used for HeartCodec inference acceleration.
RVQ
Residual Vector Quantization for discrete audio representation.
HeartCodec's key technology.
VISQOL
Virtual Speech Quality Objective Listener for audio reconstruction quality evaluation.
Used in experimental results assessment.
Open Questions Unanswered questions from this research
- 1 How to enhance the model's ability to generate complex musical styles?
- 2 How to optimize inference efficiency for real-time applications?
Applications
Immediate Applications
Short Video Background Music
Generate background music tailored to short videos with fine-grained style control.
Music Creation Tools
Assist musicians in quickly generating song segments that meet specific requirements.
Long-term Vision
Real-Time Interactive Music Creation
Develop tools supporting real-time generation and user interaction.
Abstract
We present a family of open-source Music Foundation Models designed to advance large-scale music understanding and generation across diverse tasks and modalities. Our framework consists of four major components: (1) HeartCLAP, an audio-text alignment model; (2) HeartTranscriptor, a robust lyric recognition model optimized for real-world music scenarios; and (3) HeartCodec, a low-frame-rate (12.5 Hz) yet high-fidelity music codec tokenizer that captures long-range musical structure while preserving fine-grained acoustic details and enabling efficient autoregressive modeling; (4) HeartMuLa, an LLM-based song generation model capable of synthesizing high-fidelity music under rich, user-controllable conditions (e.g., textual style descriptions, lyrics, and reference audio). In addition, it provides two specialized modes: (i) fine-grained musical attribute control, which allows users to specify the style of different song sections (e.g., intro, verse, chorus) using natural language prompts; and (ii) short, engaging music generation, which is suitable as background music for short videos. Lastly, HeartMuLa improves significantly when scaled to 7B parameters. For the first time, we show that a Suno-level, commercial-grade system can be reproduced using academic-scale data and GPU resources. We expect these foundation models to serve as strong baselines for future research and to facilitate practical applications in multimodal content production.