Leveraging Whisper Embeddings for Audio-based Lyrics Matching

The evolution of music information retrieval has transitioned from simple feature matching to sophisticated deep learning models. Early methods relied on handcrafted features and metadata, which limited accuracy. The advent of automatic speech recognition (ASR) systems like Whisper has enabled direct extraction of lyrics from audio, but transcription errors hinder performance. Recent trends focus on contrastive learning and multimodal fusion to improve robustness across languages and content types. Despite progress, existing solutions often lack transparency and scalability, especially in multilingual settings. This paper builds on these developments, proposing a fully reproducible, end-to-end audio-based lyrics matching framework that leverages Whisper embeddings, addressing the limitations of prior approaches.

Traditional lyrics matching heavily depends on accurate transcriptions, which are prone to errors, especially in musical contexts with background noise, singing variations, and multiple languages. This reliance limits scalability and cross-lingual applicability. Moreover, existing methods often involve complex pipelines with multiple stages, reducing transparency and reproducibility. The core challenge is to develop a direct audio-based semantic matching system that is robust, scalable, and capable of handling diverse content without intermediate transcription, thus overcoming the bottlenecks of current approaches.

The key innovations include: 1) exploiting Whisper decoder embeddings directly from raw audio to capture lyrics semantics, bypassing transcription errors; 2) employing contrastive learning (NT-Xent loss) to optimize the embedding space for song similarity; 3) integrating GeM pooling to enhance temporal feature aggregation; 4) analyzing multilingual capabilities of Whisper, demonstrating cross-lingual robustness; 5) implementing a multimodal fusion strategy that combines lyric-aware and audio-content models, significantly boosting retrieval accuracy. These innovations collectively set a new standard for scalable, transparent lyrics matching.

• �� Input raw audio, preprocess to mono, resample to 16kHz, truncate to 5 minutes. • Segment audio into overlapping 30-second chunks, extract log-mel spectrograms. • Use Whisper encoder-decoder to generate decoder hidden states, capturing semantic content without transcription. • Concatenate hidden states from all chunks into a sequence H. • Sample subsequences Hs of fixed length, project into a transformer encoder for contextual modeling. • Apply GeM pooling over temporal dimension to produce a fixed-size vector. • Map pooled features into a 512-dimensional semantic embedding via a linear layer. • Train with NT-Xent contrastive loss, pulling similar song pairs closer and pushing dissimilar ones apart. • During inference, compute maximum cosine similarity over overlapping chunks for song matching. • Extend with multimodal fusion by combining lyric and audio embeddings at the distance level.

The evaluation uses three datasets—SHS, DVI, LYCR—covering diverse languages and content. Baselines include TF-IDF, SBERT, and raw Whisper embeddings. Hyperparameters like subsequence length (k=1500), learning rate (1e-4), batch size (64), and training epochs (1000) are optimized with early stopping. Performance is measured by MAP, with ablation studies testing loss functions, pooling strategies, and multilingual effects. The experiments also compare unimodal and multimodal setups, validating the effectiveness of the proposed contrastive learning framework and fusion strategies in improving retrieval accuracy.

WEALY surpasses all baselines, achieving MAP scores of 0.640 (SHS), 0.328 (DVI), and 0.692 (LYCR). NT-Xent loss consistently outperforms triplet and CLEWS losses. GeM pooling and multilingual features further enhance performance. Multimodal fusion yields the highest MAP of 0.912, confirming the complementary nature of lyric and audio cues. Ablation results highlight the importance of temporal modeling and loss choice, demonstrating the robustness and scalability of the approach across datasets and languages.

This technology can be directly applied to copyright enforcement by identifying similar lyrical content across large music libraries. It also benefits music discovery platforms, enabling theme-based search without relying on transcriptions. Additionally, it supports creative workflows by finding thematically related songs, regardless of language or transcription quality, thus broadening access to diverse musical content in industry and academia.

Despite strong results, the approach's robustness decreases with noisy or distorted audio, and handling very long tracks remains challenging due to segmentation. The multimodal fusion increases computational load, limiting real-time deployment. Further work is needed to improve efficiency, noise robustness, and extend applicability to other music tasks like cover detection or genre classification.