How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation

In recent years, large language models (LLMs) have garnered significant attention for their performance in text domains. Trained on massive text corpora, these models demonstrate remarkable abilities to internalize world knowledge across diverse domains, from general reasoning to specialized technical fields. However, with the rise of multimodal learning, researchers have begun exploring the application of LLMs in audio, particularly as knowledge backbones for large audio language models (LALMs). LALMs, by pairing an audio encoder, bridge acoustic features into their pre-existing linguistic space to facilitate audio understanding. Nevertheless, existing research primarily focuses on architectural design, training strategies, or audio encoder choices, leaving the foundational role of LLMs in auditory knowledge unclear. Thus, clarifying how much auditory knowledge LLMs encode during text training and how this influences audio language models' downstream performance has become a critical research question.

The core problem is whether LLMs can effectively encode auditory knowledge through text-only training and how this knowledge impacts downstream performance in audio language models (LALMs). While LLMs excel in text domains, their application in audio remains enigmatic. Specifically, the lack of clarity regarding the foundational role of LLMs in existing LALM research makes selecting the optimal LLM as a backbone challenging. Additionally, text-trained LLMs may have inherent limitations in handling pronunciation and speech structure, which is another pressing issue.

The core innovations of this study include: 1) Introducing the AKB-2000 benchmark, covering 6 categories and 48 subcategories, to comprehensively evaluate LLMs' auditory knowledge; 2) Validating the effectiveness of auditory knowledge from text training in multimodal adaptation through cascade and audio-grounded evaluations; 3) Identifying systematic deficiencies in phonological tasks, indicating future research directions. Through these innovations, the study reveals substantial differences in auditory knowledge across LLM families and strong correlations between text-only results and audio performance, providing empirical grounding for understanding LLMs' application in audio research.

The study employs three evaluation methods to investigate LLMs' auditory knowledge:

• �� Direct probing on AKB-2000: Testing the breadth and depth of auditory knowledge, covering six categories including Music, Sound, Paralinguistic, Phonetic, Audio Quality, and Technical knowledge.

• �� Cascade evaluation: LLMs reason over audio descriptions, using an audio captioner to translate audio samples into detailed descriptions for the LLM to answer the original question.

• �� Audio-grounded evaluation: Fine-tuning each LLM into a Large Audio Language Model (LALM) with an audio encoder, using the DeSTA self-distillation framework to train, directly assessing whether inherent auditory knowledge in text-only LLMs transfers to better audio understanding after multimodal adaptation.

The experimental design includes:

• �� Datasets: Using the AKB-2000 benchmark for text evaluation, and MMAU and MMAR benchmarks for cascade and audio-grounded evaluations.

• �� Baselines: Selecting 12 open-weight LLMs, covering Qwen, Llama, Phi, and OLMo families, and 5 proprietary models as references.

• �� Metrics: Using accuracy (%) as the evaluation metric, comparing different LLMs' performance across benchmarks.

• �� Hyperparameters: In audio-grounded evaluation, using Whisper-large-v3 as the audio encoder, a 6-layer Q-Former as the modality connector, freezing both audio encoder and LLM parameters, training only the modality connector.

Results analysis shows:

• �� On the AKB-2000 benchmark, the Qwen family excels, with Qwen3-14B scoring 85.05%, significantly outperforming the highest score of 73.45% from the Llama family.

• �� Cascade evaluation shows that a simple pipeline using text descriptions can match or even surpass several state-of-the-art end-to-end LALMs, indicating that current systems are bottlenecked by the audio encoder.

• �� Audio-grounded evaluation shows that auditory knowledge from text training effectively transfers to multimodal adaptation, with Qwen3-14B performing well on audio inputs in the MMAU and MMAR benchmarks.

Application scenarios include:

• �� Audio understanding systems: Enhancing system performance by selecting the optimal LLM as the LALM backbone, applicable to speech recognition, music recommendation, etc.

• �� Multimodal learning: Verifying the transferability of text-trained knowledge across different modalities, providing insights for other multimodal domains such as image-text joint learning.

• �� Audio encoder optimization: Identifying current system bottlenecks, driving the development of more powerful audio encoders to enhance overall LALM performance.

Limitations and outlook include:

• �� Assumptions: The study assumes text benchmarks can serve as reliable proxies for selecting backbone models, which may not fully capture LLMs' performance in real audio scenarios.

• �� Failure scenarios: Performance on phonological tasks is generally low, indicating inherent limitations of text-trained LLMs in handling pronunciation and speech structure.

• �� Computational costs: The audio encoder used in the study may limit the overall performance of the LALM, suggesting the need for exploring more powerful encoders in the future.