Masked diffusion enables coherent beat tracking
Masked Diffusion model improves beat tracking by modeling multiple plausible outputs, reducing incoherence and enhancing stability.
Key Findings
Methodology
This paper introduces a Masked Diffusion Model (MDM) tailored for beat tracking, employing independent masking of beats and downbeats, balanced inference scheduling, and peak detection. During training, the model masks beat and downbeat labels independently, optimizing a diffusion-based objective that predicts masked tokens iteratively. In inference, the model starts from a fully masked sequence, progressively unmasking tokens over multiple steps guided by confidence scores, enabling the selection of coherent beat hypotheses. The approach leverages multi-task learning, combining a shift-tolerant BCE loss with confidence-based masking, and incorporates ensemble techniques to improve robustness. Experimental results on GTZAN demonstrate significant improvements in F1, CMLt, and AMLt metrics, outperforming baseline models and reducing erratic beat predictions.
Key Results
- On GTZAN, the 8-step inference achieved a beat F1 score of 89.7±0.2, outperforming previous models by approximately 3%. Downbeat F1 reached 79.5±0.3, with CMLt and AMLt scores of 82.9±0.4 and 92.5±0.2 respectively. The model notably reduced tempo doubling/halving errors and consecutive downbeats, indicating higher temporal coherence. Ensembling further boosted performance, with metrics stabilizing across diverse musical styles.
- Compared to traditional autoregressive or single-step models, the MDM effectively captures multiple plausible beat grids, handling ambiguous and complex rhythms. The iterative inference process, guided by confidence scores, mitigates label confusion and reduces false positives. Ablation studies confirmed the importance of independent masking, peak detection, and confidence scheduling for optimal results.
- The experimental setup involved training on 4556 tracks, validating hyperparameters, and testing on GTZAN. The model's robustness was validated through cross-validation, ablation, and ensemble experiments. Results consistently showed superior stability and accuracy, especially in challenging musical contexts with tempo variations and time signature changes.
Significance
This work marks a significant advance in beat tracking, addressing longstanding issues of incoherent outputs in neural models. By modeling multiple plausible beat hypotheses and enabling iterative, confidence-guided inference, the approach enhances both accuracy and robustness. It reduces reliance on post-processing, broadening applicability to diverse musical genres and complex rhythmic structures. The methodology paves the way for more reliable music analysis tools, facilitating applications in automatic transcription, real-time accompaniment, and music generation. The integration of diffusion-based generative modeling into temporal sequence prediction opens new research avenues for music AI, bridging the gap between probabilistic modeling and structured output generation.
Technical Contribution
The paper pioneers the adaptation of Masked Diffusion Models (MDMs) for beat tracking, introducing key innovations such as independent masking of beat/downbeat labels, a balanced inference schedule, and peak suppression techniques. These modifications enable the model to handle highly imbalanced label distributions and multi-task learning challenges. The diffusion process iteratively refines predictions, allowing the model to select valid beat hypotheses early in the process, reducing incoherence. The approach combines the strengths of diffusion models with traditional beat tracking architectures, maintaining frame-level encoding while supporting multi-modal, multi-hypothesis output generation. This represents a fundamental shift from deterministic, single-output models to probabilistic, multi-hypothesis frameworks in music temporal modeling.
Novelty
This study is the first to apply Masked Diffusion modeling to beat tracking, innovatively enabling the modeling of multiple plausible beat grids within a single framework. Unlike prior autoregressive or single-step classifiers, the proposed method supports iterative, confidence-guided unmasking, effectively capturing the inherent ambiguity in musical rhythm. The independent masking of beats and downbeats, combined with a balanced unmasking schedule and peak suppression, constitutes a novel architecture tailored for music signals. These innovations collectively address the core challenge of incoherent and erratic beat predictions, setting a new benchmark in the field.
Limitations
- Inference speed remains a concern, especially with multiple ensemble models and increased inference steps, limiting real-time deployment. Computational costs are higher than traditional models, requiring optimization for practical use.
- The model's performance degrades in extremely complex or noisy environments, such as highly irregular rhythms or recordings with significant background interference. Further robustness improvements are needed.
- Training demands large annotated datasets and substantial computational resources, which may hinder adaptation to low-resource settings or less-studied musical genres.
Future Work
Future research will focus on reducing inference latency, possibly through model pruning or distillation. Extending the framework to handle more complex rhythmic structures, such as polymeters and irregular time signatures, is also a priority. Incorporating self-supervised learning and multi-modal inputs like MIDI or symbolic data could enhance generalization. Additionally, exploring real-time implementation and deployment in live performance scenarios will be crucial for practical applications. Theoretical analysis of diffusion dynamics in temporal modeling may further deepen understanding and improve model design.
AI Executive Summary
Music beat tracking has long been a challenging task due to the ambiguity and complexity of rhythmic structures. Traditional neural network models, which predict beat and downbeat locations in a single step, often produce inconsistent or erratic outputs, especially in music with variable tempo or multiple plausible beat interpretations. These issues hinder the development of reliable automatic music analysis tools, limiting their application in real-world scenarios.
This paper introduces a novel approach based on Masked Diffusion Models (MDMs), which fundamentally change how beat predictions are generated. Instead of a single prediction, the model employs an iterative inference process, gradually revealing beat hypotheses guided by confidence scores. During training, the model learns to predict masked beat and downbeat labels independently, using a specialized loss function that sharpens peak predictions and balances label imbalances. In inference, the model starts from a fully masked sequence and progressively unmask the most confident predictions, employing a balanced schedule to prevent bias toward non-events. Peak detection is applied after each step to ensure musical coherence.
Experimental results on the GTZAN dataset demonstrate that this approach significantly outperforms existing models, achieving a beat F1 score of 89.7% and reducing tempo inconsistencies. The model's ability to handle multiple plausible interpretations and produce stable, coherent outputs marks a major step forward in music information retrieval. The use of ensemble techniques further enhances robustness, making the system suitable for diverse musical styles.
While the method improves accuracy and reduces erratic behavior, it introduces increased computational costs and longer inference times, posing challenges for real-time applications. Nonetheless, this work opens new avenues for integrating generative diffusion techniques into temporal sequence modeling, promising broader impacts in automatic music transcription, live accompaniment, and music generation. Future efforts will aim to optimize speed, extend to more complex rhythms, and incorporate multi-modal data, pushing the boundaries of AI-driven music analysis.
Deep Dive
Abstract
Current neural networks for beat tracking generate invalid outputs, such as consecutive downbeats and erratic tempo changes, even when these are not present in the training data. Heavy post-processing techniques can alleviate these problems, but the original cause of this inconsistent behaviour remains unknown. We hypothesise that it stems from inadequate modelling of multiple plausible output beat grids, resulting in an invalid mixture of competing interpretations. We propose a masked diffusion approach that properly models multiple outputs and enables the model to build coherent predictions through iterative inference. We devise three modifications to standard masked diffusion that enable its application to beat tracking: independent masking of beats and downbeats during training and inference, a balanced masking scheduler for inference, and peak-picking across inference steps. Our approach reduces erratic behaviours and improves beat-tracking performance.