Pengi: An Audio Language Model for Audio Tasks
Pengi unifies audio tasks as text generation, reaching 0.4667 SPIDEr on AudioCaps and 0.9195 accuracy on ESC50.
Key Findings
Methodology
Pengi uses an HTSAT audio encoder and a CLIP text encoder, followed by two 8-layer Transformer mapping networks that produce 40-token continuous prefixes. Their concatenation forms an 80-token prefix for frozen GPT2-base (124M). Classification, retrieval, captioning, and question answering are reformulated as (audio,text)-to-text generation. Cross-entropy trains the audio encoder and mappers, while GPT2 and the text encoder remain frozen; beam search uses width 5.
Key Results
- Pengi obtains SPIDEr scores of 0.4667 on AudioCaps and 0.2709 on Clotho, corresponding to relative gains of 6.6% and 26% over previous best results. ClothoAQA accuracy reaches 64.5%, a 1.5% relative gain over the supervised benchmark.
- Zero-shot ESC50 accuracy is 0.9195, exceeding AudioCLIP, CLAP, and LAION-CLAP by approximately 32%, 11%, and 1%, respectively. FSD50K mAP is 0.4676 and UrbanSound8K accuracy is 0.7185.
- Multitask training improves captioning over caption-only training: AudioCaps SPIDEr rises from 0.4551 to 0.4667 and Clotho from 0.2648 to 0.2709, supporting shared representation transfer.
Significance
Pengi unifies audio classification, retrieval, captioning, and question answering through one generative interface. This reduces task-specific heads and fine-tuning pipelines while allowing natural-language outputs. More broadly, it demonstrates that a frozen language model can acquire useful audio understanding through learned continuous prefixes, offering a practical route toward general-purpose audio assistants and cross-modal search.
Technical Contribution
The central engineering contribution is audio-conditional prefix tuning. HTSAT and CLIP representations are converted by trainable mappers into GPT2-compatible continuous prefixes, while GPT2 stays frozen. Eight instruction-style templates cover captioning, AQA, events, scenes, emotion, music, notes, and metadata. A 3.4-million-pair mixture provides multitask transfer. Log-likelihood and text-embedding matching connect free-form generation to closed-set evaluation.
Novelty
According to the authors, Pengi is the first audio architecture to support open-ended and closed-ended tasks with one model, one captioning-style objective, and no task-specific extensions. Unlike CLAP, Mulan, and LAION-CLAP, it can generate unrestricted language; unlike prior generative audio systems, it also handles classification, regression, and retrieval through textual prompts.
Limitations
- The frozen GPT2-base and fixed 80-token prefix constrain reasoning and linguistic capacity. Randomly truncating audio to seven seconds can remove long-range structure, rare events, temporal order, or multi-speaker context.
- Although training uses 3.4 million pairs, source datasets and templates are imbalanced. Text matching can penalize valid synonyms or verbose answers, so closed-set scores may underestimate semantic correctness.
- The study does not systematically evaluate larger language models, long recordings, multilingual behavior, robustness, fairness, or deployment latency.
Future Work
Promising directions include larger instruction-tuned language models, long-audio chunking with temporal localization, richer and multilingual templates, few-shot adaptation, and calibrated generation-to-label matching. Future benchmarks should test real noise, long context, safety, semantic equivalence, and computational efficiency rather than relying mainly on exact or embedding-based label matching.
AI Executive Summary
Audio understanding has traditionally been divided into classification, retrieval, captioning, and question answering. Contrastive systems such as CLAP, Mulan, and LAION-CLAP offer strong zero-shot closed-set prediction, but they cannot naturally produce explanations or captions. Generative systems address open-ended language, yet are rarely evaluated across closed-set tasks. Pengi targets this gap.
The model reformulates every task as audio plus text prompt to text output. HTSAT encodes audio, CLIP encodes the prompt, and two 8-layer Transformer mappers convert them into 40 embeddings each. The resulting 80-vector prefix conditions a frozen GPT2-base. Training updates only the audio-side encoder and mappers, using 3.4 million audio-text pairs, eight instruction templates, and autoregressive cross-entropy. Inference uses beam search with width 5.
Across 21 downstream tasks, Pengi reaches SPIDEr scores of 0.4667 on AudioCaps and 0.2709 on Clotho, relative gains of 6.6% and 26%; ClothoAQA accuracy is 64.5%. Zero-shot ESC50 accuracy reaches 0.9195, outperforming several CLAP-family baselines. Multitask learning also improves captioning. Limitations include seven-second truncation, the modest frozen GPT2, data imbalance, and imperfect text matching. Nevertheless, Pengi provides a compelling blueprint for connecting audio encoders with language models for general-purpose audio understanding.
Deep Analysis
Background
Self-supervised, supervised, and contrastive transfer learning have produced strong audio representations. CLAP and related models align audio with text for zero-shot classification and retrieval, while AudioCaps and Clotho support caption generation. In NLP, T5 and FLAN showed that instruction-formatted text-to-text learning can unify tasks; Frozen and Flamingo extended language models to vision. Audio lacked a comparable architecture that handled both open-ended and closed-ended tasks without separate modules.
Core Problem
The target is a single system that accepts arbitrary audio and a natural-language prompt, then generates text for captioning, AQA, classification, regression, or retrieval. Audio is a long continuous signal, labels and answer formats vary, and language models have no native audio interface. Moreover, semantically correct free-form answers may not exactly match fixed labels, complicating evaluation.
Innovation
- ��Unified formulation: all tasks become (audio,text)-to-text generation.
- ��Continuous prefix interface: HTSAT and CLIP outputs are mapped into GPT2 prefixes instead of modifying GPT2.
- ��Instruction templates: eight templates adapt FLAN-style instruction tuning to audio.
- ��Unified closed-set evaluation: candidate log-likelihood or cosine similarity between generated text and label embeddings.
- ��Large multitask mixture: 3.4 million pairs jointly train open- and closed-ended behavior.
Methodology
- ��Input: audio is sampled at 44.1 kHz and converted to 64-bin log-Mel spectra with 320 ms hop, 1024 ms window, and 50–8000 Hz range; files are randomly truncated to seven seconds.
- ��Encoding: HTSAT produces audio features; a frozen CLIP text encoder processes prompts up to length 40.
- ��Mapping: two 8-layer Transformer networks each generate 40 embeddings, yielding an 80-vector concatenated prefix.
- ��Generation: frozen GPT2-base autoregressively predicts target text.
- ��Optimization: Adam for 60 epochs, batch size 384 on 20 V100 GPUs, learning rate 1e-4, 2,000 warm-up steps; cross-entropy updates the audio encoder and mappers.
- ��Inference: beam size 5; closed-set prediction uses log-likelihood or text-embedding cosine matching.
Experiments
Training combines AudioSet, FSD50K, CochlScene, MSP Podcast, CMU MOSI/MOSEI, MELD, NSynth, FMA, AudioCaps, ClothoV2, ClothoAQA, and auxiliary metadata sources, totaling 3.4 million pairs. Evaluation covers 21 tasks, including ESC50, UrbanSound8K, DCASE2017, TUT2017, CREMA-D, RAVDESS, Vocal Sound, NSynth, and Clotho retrieval. Metrics include SPIDEr, accuracy, mAP, F1, and R@1. Baselines include CLAP and supervised systems; an ablation compares caption-only against multitask training.
Results
Pengi reaches SPIDEr 0.4667 on AudioCaps and 0.2709 on Clotho, versus best supervised scores of 0.438 and 0.215. ClothoAQA accuracy is 64.5%. Closed-set results include 0.9195 accuracy on ESC50, 0.4676 mAP on FSD50K, 0.7185 accuracy on UrbanSound8K, and 0.3525 on TUT2017. Clotho text-to-audio retrieval achieves R@1/R@5/R@10 of 9.4/26.1/36.7. Multitask training consistently improves captioning.
Applications
Pengi can support natural-language audio search, environmental sound summaries, smart-home event detection, accessibility alerts, media annotation, and music analysis. Open-ended deployment needs prompt design and safety filtering; strict classification additionally needs a label vocabulary, text-matching calibration, and confidence thresholds. Industry value comes from reducing separate models and enabling one conversational interface across audio domains.
Limitations & Outlook
Seven-second truncation limits long events, temporal ordering, and weak-source reasoning. Frozen GPT2-base restricts complex reasoning and may reduce fluency; the 3.4-million-pair mixture is not necessarily balanced across speech, music, and environmental sound. Generated text may be semantically correct but fail label matching. Further work should address long context, larger models, multilingual and noisy conditions, robustness, fairness, privacy, and inference cost.
Plain Language Accessible to non-experts
Imagine Pengi as a restaurant with one universal kitchen. A customer gives the front desk a sound and an instruction, such as “What happened?” or “Describe this recording.” One worker listens to the sound and writes an internal note; another turns the customer’s request into a clear order. The two notes are placed together and handed to a chef who is already excellent at writing, although the chef cannot directly hear the original sound.
The kitchen does not build a separate appliance for every dish. Captioning, question answering, classification, and search all use the same workflow; only the customer’s wording changes. During training, the restaurant studies about 3.4 million sound-and-text examples and learns how sound notes should guide written answers.
There are practical catches. The restaurant usually listens to only seven seconds, so it may miss what happened before or after. If the chef writes “canine” while the menu says “dog,” a strict checker may mark the answer wrong. Even so, Pengi shows how one shared counter can connect sounds and language instead of forcing every task into a separate system.
ELI14 Explained like you're 14
Think of a superpower in a video game: your computer hears a dog bark, a train, or a song and explains what is going on. Older systems often had one special button for each job—one for recognizing dogs, one for identifying places, and another for writing captions. Changing the subject meant training a new machine, like needing a different console for every school class.
Pengi uses “task cards.” You give it a sound plus a request, such as “describe this” or “what instrument is playing?” It turns the sound into a secret sequence of clues and places those clues before GPT2, a model that is good at writing. GPT2 does not literally have ears, but training teaches it how sound clues should lead to words.
Researchers trained it with roughly 3.4 million examples from datasets such as AudioSet, AudioCaps, and Clotho. The results are impressive: AudioCaps captioning reaches SPIDEr 0.4667, ESC50 classification reaches 91.95% accuracy, and Clotho question answering reaches 64.5%.
It is not magic, though! It usually hears only seven seconds, so a longer story can be confusing. It may say a correct synonym that a strict score does not recognize. The next upgrade would let it listen longer, handle noisy recordings, and explain exactly when an event happened.
Glossary
Audio Language Model
A model that uses audio as conditioning information for language generation. It can express both open-ended answers and closed-set predictions in text.
Pengi is introduced as an Audio Language Model.
HTSAT
A Transformer-based audio encoder that converts log-Mel spectrograms into learned audio representations. It serves as Pengi’s audio front end.
HTSAT produces the audio embedding before prefix mapping.
Prefix tuning
A method that maps external information into continuous vectors placed before a language-model sequence. The language model can remain frozen while the prefix-producing modules learn.
Pengi uses audio and text prefixes to condition GPT2.
CLAP
Contrastive Language-Audio Pretraining aligns audio and text embeddings so matching pairs are close in representation space. It is strong for zero-shot scoring but does not natively generate free-form text.
Pengi uses a CLAP-derived audio backbone and compares against CLAP.
SPIDEr
An audio-captioning metric combining CIDEr and SPICE. It measures lexical overlap and semantic or structural similarity.
It ranks Pengi on AudioCaps and Clotho captioning.
Open Questions Unanswered questions from this research
- 1 Long-audio understanding remains unresolved: seven-second truncation can hide event order and context. Progress requires chunk memory, temporal localization, and long-context language models.
- 2 Reliability, hallucination, privacy, and safety are not systematically tested, especially for surveillance, healthcare, children, and high-stakes environments. A calibrated human-audited evaluation is needed.
Applications
Immediate Applications
Natural-language audio search
Media libraries and monitoring platforms can generate descriptions and labels, then retrieve clips using text embeddings. Deployment requires domain audio, a controlled label vocabulary, and similarity thresholds to limit false matches.
Accessibility sound alerts
A device can convert doorbells, alarms, vehicles, or nearby speech into short text notifications. Noise reduction, confidence thresholds, and human confirmation are important when missed events could be dangerous.
Long-term Vision
General-purpose audio assistant
A future assistant could jointly understand speech, environmental sounds, and music, then answer questions such as “What happened a moment ago?” Key obstacles include long-term memory, multilingual support, privacy, safety, and low-latency inference.
Abstract
In the domain of audio processing, Transfer Learning has facilitated the rise of Self-Supervised Learning and Zero-Shot Learning techniques. These approaches have led to the development of versatile models capable of tackling a wide array of tasks, while delivering state-of-the-art performance. However, current models inherently lack the capacity to produce the requisite language for open-ended tasks, such as Audio Captioning or Audio Question & Answering. We introduce Pengi, a novel Audio Language Model that leverages Transfer Learning by framing all audio tasks as text-generation tasks. It takes as input, an audio recording, and text, and generates free-form text as output. The input audio is represented as a sequence of continuous embeddings by an audio encoder. A text encoder does the same for the corresponding text input. Both sequences are combined as a prefix to prompt a pre-trained frozen language model. The unified architecture of Pengi enables open-ended tasks and close-ended tasks without any additional fine-tuning or task-specific extensions. When evaluated on 22 downstream tasks, our approach yields state-of-the-art performance in several of them. Our results show that connecting language models with audio models is a major step towards general-purpose audio understanding