Evaluation of pretrained language models on music understanding

Recent years have seen significant advancements in Music Information Retrieval (MIR), particularly in audio-to-text and text-to-audio retrieval. The introduction of multimodal systems has enabled applications such as music generation and captioning. However, despite these practical successes, the evaluation of large language models' (LLM) musical knowledge remains limited. Existing research primarily focuses on semantic similarity evaluations for language models, with relatively little attention given to the music domain.

The core problem is the inadequate evaluation of existing large language models' musical knowledge, particularly their poor performance in handling negation and specific words. Addressing this issue is crucial for improving the accuracy and robustness of music information retrieval applications. Due to the complexity and diversity of the music domain, existing semantic similarity evaluation methods struggle to effectively capture musical knowledge.

The core innovation of this paper is the proposal of a triplet evaluation method based on Audioset ontology to quantify language models' musical knowledge. By generating triplets consisting of an anchor, a positive, and a negative label, the study can more accurately reveal deficiencies in music understanding. Unlike traditional semantic similarity evaluations, this method effectively captures the complexity of the music domain.

• �� Generate triplets using Audioset ontology, consisting of an anchor, a positive, and a negative label.
• �� Quantify models' musical knowledge by comparing the relative similarity between anchor-positive and anchor-negative labels.
• �� Evaluate six general-purpose Transformer models on music triplets.
• �� Analyze models' sensitivity to negation and specific words.

The experimental design includes 13,633 music genre triplets and 37,640 instrument triplets generated from Audioset ontology. Evaluated models include MPNet, DistilRoBERTa, MiniLM, and ALBERT. Models' musical knowledge is assessed by comparing the relative similarity between anchor-positive and anchor-negative labels. Experiments also include sensitivity analysis to different prompts and evaluation of negation handling capabilities.

Results show that despite high accuracy on music triplets, models exhibit inconsistencies in handling negation and specific words. For instance, MPNet achieved 71.3% accuracy on instrument triplets, dropping to 41.1% when handling negation. Accuracy improved when using definitions instead of labels, e.g., Para-MiniLM-L3 achieved 81.6% accuracy on instrument definitions.

Application scenarios include music information retrieval, music generation, and captioning. Improving models' musical knowledge can enhance the accuracy and robustness of these applications. Additionally, the findings can guide future model improvements, particularly in handling negation and specific words.

Limitations include poor model performance with negation, with accuracy below random chance. Triplet generation relies on Audioset ontology, which may introduce bias. The study focuses on Western music, possibly limiting applicability to other music types. Future research could explore using more complex ontology structures like WordNet to improve triplet generation.