Dialogue-Aware Video-to-Music Generation Using Public Domain Film Collections
Introduced dialogue-aware video-to-music generation using OSSL-v2 dataset, improving temporal fidelity of generated music.
Key Findings
Methodology
The study proposes a novel video-to-music generation method using dialogue from public domain films as a conditioning signal. Specifically, it extends existing models' video cross-attention with a time axis and modulates it frame-by-frame with the dialogue track. This approach captures the temporal coupling between film music and on-screen speech more effectively.
Key Results
- GVMGen achieved the best performance on the OSSL-v2 test set, particularly in paired fidelity.
- The dialogue adapter significantly reduced KL divergence for VidMuse on the OES-Com dataset from 1.47 to 0.85.
- On low-residual-music subsets, the introduction of dialogue signals still improved paired fidelity, indicating its contribution is not reliant on music leakage.
Significance
This research addresses the instability of datasets in the video-to-music generation field by providing a large-scale, self-hosted OSSL-v2 dataset. By introducing dialogue as a conditioning signal, it demonstrates the importance of dialogue in film music generation. This method not only enhances the temporal fidelity of generated music but also provides a stable benchmark for future research.
Technical Contribution
Technical contributions include the introduction of dialogue signals as conditioning input, enhancing the temporal fidelity of video-to-music generation models. By modulating the dialogue track frame-by-frame, this approach outperforms existing baselines across different datasets, especially in paired fidelity.
Novelty
This study is the first to incorporate dialogue signals into the video-to-music generation process, significantly improving the temporal fidelity of generated music. The innovation lies in leveraging the temporal coupling between dialogue and music rather than relying solely on global video information.
Limitations
- The method is limited in handling videos without dialogue, as it lacks conditioning input from dialogue signals.
- The quality of dialogue signals may affect the final music generation outcome.
Future Work
Future research could explore enhancing model generation capabilities in the absence of dialogue signals and further optimizing the extraction and utilization of dialogue signals.
AI Executive Summary
Video-to-music generation is a crucial research area aimed at automatically generating music that matches video content. However, existing methods often rely on unstable web datasets, leading to poor reproducibility. To address this, the research team introduced a new self-hosted dataset, OSSL-v2, containing 34,343 video clips totaling 246.4 hours, all sourced from public domain films.
The core innovation of this study is the use of dialogue signals as conditioning input to enhance the temporal fidelity of video-to-music generation models. By incorporating a time axis and modulating the dialogue track frame-by-frame, the study demonstrates how the temporal coupling between dialogue and music influences the quality of generated music. Experimental results show that this method outperforms existing baselines across multiple datasets, particularly in paired fidelity.
While the method excels in videos with dialogue, it is limited in scenarios without dialogue signals. Future research could explore enhancing model generation capabilities in the absence of dialogue signals and further optimizing the extraction and utilization of dialogue signals.
Deep Dive
Abstract
Video-to-music generation has drawn growing interest for its role in conveying the emotion of visual media, including film. Progress in the field, however, is hampered by a reproducibility gap: models are often trained on crawled corpora referenced through YouTube URLs that may be deleted, with the underlying data often difficult and time-consuming to retrieve. To address this, we introduce the Open Screen Soundtrack Library version 2 (OSSL-v2), a self-hosted corpus of 34,343 video clips totaling 246.4 hours, sourced from public-domain films. Unlike crawled corpora, OSSL-v2 is reproducible (i.e., not subject to link rot) and copyright-conscious, yet still large enough to train functional video-to-music models. We then use this film-domain corpus to study dialogue as a conditioning signal for video-to-music generation, motivated by the close temporal coupling between film music and on-screen speech. Specifically, we augment existing models' video cross-attention with a time axis and modulate it frame-by-frame with the dialogue track. Evaluated on both public-domain and commercial films, our approach shows improvement over the state-of-the-art baselines. The dataset is available at https://huggingface.co/datasets/McAuley-Lab/OSSL-v2.