MPEcho: A Melody and Phoneme-Aware Generative Framework for Controllable Cover Song Generation
MPEcho extends SongEcho with a phoneme encoder and length regulator, reducing phoneme error rate to 18.65%.
Key Findings
Methodology
MPEcho integrates a phoneme encoder and length regulator to provide precise phoneme-level temporal boundaries, improving lyrical accuracy. Phonsa, a Whisper-based transcription model, enhances phoneme annotation quality.
Key Results
- Phoneme error rate (PER) reduced from SongEcho's 45.62% to 18.65%, significantly improving lyrical consistency.
- Phonsa achieves 32.6ms mean absolute error (MAE) in phoneme alignment, outperforming MFA's 233.9ms.
- MPEcho scored higher in melody consistency and vocal naturalness in subjective tests.
Significance
MPEcho is the first end-to-end framework combining melody and phoneme control for cover song generation, addressing long-standing issues in lyrical accuracy and melody alignment.
Technical Contribution
Introduced SVS-inspired phoneme-level timing control, improving lyrical precision. Developed Phonsa to overcome audio-phoneme alignment bottlenecks.
Novelty
MPEcho is the first framework to combine melody and phoneme control, significantly enhancing the precision of lyrical rendering compared to prior models.
Limitations
- Phonsa struggles with boundary detection in complex polyphonic environments.
- MPEcho heavily relies on annotation quality; errors can degrade generation quality.
- High computational cost limits scalability.
Future Work
Future work includes exploring multilingual phoneme annotation and optimizing computational efficiency for real-time generation.
AI Executive Summary
Cover song generation requires preserving the melody and lyrics of a reference song while creatively reinterpreting other musical components. Existing models like SongEcho achieve melody control but suffer from high phoneme error rates (45.62%).
MPEcho introduces a phoneme encoder and length regulator, combining melody and phoneme control to reduce phoneme error rate to 18.65%. Additionally, Phonsa, a Whisper-based transcription model, provides high-precision phoneme annotations for training and inference.
Experimental results show MPEcho outperforms existing models in melody consistency and vocal naturalness, demonstrating the potential of SVS techniques in full-song generation. Despite challenges in boundary detection and computational efficiency, this framework represents a significant advancement in music generation technology.
Deep Analysis
Background
Cover song generation is an emerging task in generative AI, requiring melody preservation and creative reinterpretation. Existing models like SongEcho achieve melody control but lack fine-grained phoneme accuracy.
Core Problem
Current models fail to accurately render lyrics due to the absence of phoneme-level timing control, resulting in high phoneme error rates and degraded generation quality.
Innovation
MPEcho combines melody and phoneme control using a phoneme encoder and length regulator for precise timing. Phonsa addresses annotation bottlenecks by providing high-quality phoneme boundaries.
Methodology
- �� Phoneme encoder extracts phoneme-level embeddings.
- �� Length regulator aligns phoneme embeddings with audio temporal axis.
- �� Phonsa transcription model improves phoneme annotation accuracy.
- �� Multi-condition generation integrates melody and phoneme controls.
Experiments
Experiments use a Chinese pop song dataset, evaluating metrics like PER, RPA, MAE, and conducting subjective listening tests.
Results
MPEcho reduces PER to 18.65%, outperforming SongEcho's 45.62%. Phonsa achieves 32.6ms MAE in phoneme alignment, far better than MFA's 233.9ms.
Applications
Applicable to cover song generation, music education, and personalized music creation, especially where high lyrical accuracy is required.
Limitations & Outlook
Phonsa struggles with boundary detection in complex environments. High computational cost limits scalability.
Plain Language Accessible to non-experts
Think of MPEcho as a smart music studio. Melody control acts like a conductor ensuring the generated music follows the original tune. Phoneme control works like a lyric editor, precisely adjusting pronunciation and timing. Phonsa is like a professional sound engineer marking every note and lyric's timing, ensuring perfect synchronization.
ELI14 Explained like you're 14
Imagine you're playing a music game. MPEcho is like a super helper that lets you remix your favorite songs while keeping the lyrics perfectly in sync! Phonsa is like the game's rhythm marker, making sure every note and word hits the right beat. Cool, right?
Glossary
Phoneme Encoder
Extracts phoneme-level embeddings for precise lyrical rendering.
Used for phoneme-level timing control in generation.
Length Regulator
Expands phoneme embeddings to match the audio temporal axis.
Ensures accurate phoneme alignment.
Phoneme Error Rate (PER)
Measures the accuracy of generated lyrics compared to reference lyrics.
Evaluates lyrical precision in generation.
Phonsa
A Whisper-based transcription model optimizing phoneme annotation.
Provides phoneme boundaries for training and inference.
Raw Pitch Accuracy (RPA)
Measures pitch accuracy of generated music relative to reference melody.
Evaluates melody control effectiveness.
Open Questions Unanswered questions from this research
- 1 How to improve phoneme boundary detection in polyphonic environments?
- 2 Can phoneme annotation be extended to multilingual datasets?
Applications
Immediate Applications
Cover Song Generation
Music creators can use MPEcho to produce high-quality covers while preserving melody and lyrics.
Music Education
Helps students learn the integration of lyrics and melody through precise generation.
Long-term Vision
Personalized Music Creation
Future systems could allow users to input lyrics and melodies for fully customized music.
Abstract
Cover song generation (CSG) should preserve the melodic and linguistic content of a reference song while recreating the remaining musical components. The state-of-the-art model SongEcho utilizes $F_0$ sequences and voiced/unvoiced (V/UV) tags for conditioning; however, implicit linguistic information from V/UV tags cannot guarantee lyric accuracy, leading to a high phoneme error rate (PER). Inspired by singing voice synthesis (SVS), we propose MPEcho, which integrates a phoneme encoder and a length regulator (LR) into the SongEcho framework. By providing explicit phoneme-level conditioning and precise temporal boundaries, MPEcho significantly reduces PER. To enable this, we developed Phonsa, a Whisper-based automatic transcription model that provides high-precision phoneme-level annotations for singing voices, overcoming the scarcity of high-quality audio-phoneme pairs. Experimental results validate the effectiveness of Phonsa for alignment and MPEcho for end-to-end CSG. The audio samples, code and weights can be accessed from https://lonian6.github.io/MPEcho.github.io/.