Source-Adaptive Data Curation for Bilingual NVV-Aware ASR
Proposed a source-adaptive data curation strategy, enhancing bilingual NVV-aware ASR performance, improving final score from 33.32 to 53.61.
Key Findings
Methodology
The study introduces an NVV-aware ASR system based on Whisper-medium, achieving joint decoding of lexical and NVV tags through checkpoint-compatible vocabulary remapping. The system employs a source-adaptive data curation strategy, refining public NVV corpora with acoustic augmentation and multimodal LLM filtering while mining spontaneous NVVs from in-the-wild media.
Key Results
- Under the official bilingual evaluation protocol, the system's final score improved from 33.32 to 53.61, indicating significant enhancement in NVV event recognition and relative localization.
- In Mandarin, F1micro increased from 0.3325 to 0.5666, and mNTD decreased from 0.6602 to 0.4443.
- In English, F1micro increased from 0.2964 to 0.5081, and mNTD decreased from 0.7459 to 0.4357.
Significance
This research holds significant implications for academia and industry, addressing the loss of nonverbal vocalization information in traditional ASR systems, thereby enhancing dialogue understanding and media analysis capabilities. By introducing NVV-awareness, the system achieves more accurate speech recognition and event localization in bilingual settings.
Technical Contribution
Technical contributions include a novel NVV-aware decoding method that achieves joint generation of lexical and NVV tags through vocabulary remapping. The data curation strategy combines acoustic augmentation and multimodal filtering to improve the diversity and annotation consistency of existing corpora.
Novelty
This study is the first to achieve NVV-aware joint decoding in a bilingual setting, innovatively introducing a source-adaptive data curation strategy that significantly enhances speech recognition accuracy and event localization capabilities.
Limitations
- The system experiences some lexical accuracy degradation in English speech recognition, requiring further optimization.
- Certain fine-grained NVV categories remain challenging to recognize, possibly needing more targeted supervision.
Future Work
Future work will focus on improving language balance, enhancing English lexical recognition robustness, and reducing the trade-off between NVV recognition and lexical accuracy.
AI Executive Summary
Nonverbal vocalizations (NVVs) like laughter and sighs convey affective and interactional information in conversations, often overlooked by traditional ASR systems. This paper presents a bilingual NVV-aware ASR system that achieves joint decoding of lexical and NVV tags through checkpoint-compatible vocabulary remapping. The system employs a source-adaptive data curation strategy, refining public NVV corpora with acoustic augmentation and multimodal LLM filtering while mining spontaneous NVVs from in-the-wild media.
In experiments, the system's final score improved from 33.32 to 53.61 under the official bilingual evaluation protocol, indicating significant enhancement in NVV event recognition and relative localization. F1micro for Mandarin and English increased to 0.5666 and 0.5081, respectively, while mNTD decreased to 0.4443 and 0.4357.
Despite significant progress, the system experiences some lexical accuracy degradation in English speech recognition, requiring further optimization. Future work will focus on improving language balance, enhancing English lexical recognition robustness, and reducing the trade-off between NVV recognition and lexical accuracy.
Deep Analysis
Background
Nonverbal vocalizations (NVVs) convey affective and interactional information in conversations, often overlooked by traditional ASR systems. Recent datasets have expanded coverage of nonverbal speech, such as ESC and VocalSound, but existing resources exhibit heterogeneous annotation quality, category imbalance, and limited acoustic diversity.
Core Problem
Traditional ASR systems typically simplify NVVs into non-speech symbols, losing important information for dialogue understanding and media analysis. This issue is particularly pronounced in bilingual settings, where speaking styles, acoustic realizations, and annotation distributions differ significantly across languages and sources.
Innovation
The paper introduces an NVV-aware ASR system based on Whisper-medium, achieving joint decoding of lexical and NVV tags through checkpoint-compatible vocabulary remapping. The system employs a source-adaptive data curation strategy, refining public NVV corpora with acoustic augmentation and multimodal LLM filtering while mining spontaneous NVVs from in-the-wild media.
Methodology
- �� NVV-aware decoding: Achieves joint generation of lexical and NVV tags through vocabulary remapping.
- �� Data curation strategy: Combines acoustic augmentation and multimodal LLM filtering to refine public NVV corpora.
- �� Spontaneous NVV mining: Automates preprocessing and annotation of spontaneous NVVs from in-the-wild media.
Experiments
Experiments were conducted under the official bilingual evaluation protocol, using Clean Public NVV Data and Movie NVV Data for training, and evaluated on the Final Stage test set. The system achieved significant improvements in F1micro and mNTD for both Mandarin and English.
Results
The system's final score improved from 33.32 to 53.61 under the official bilingual evaluation protocol. F1micro for Mandarin and English increased to 0.5666 and 0.5081, respectively, while mNTD decreased to 0.4443 and 0.4357.
Applications
The system can be applied in dialogue understanding, media analysis, and computational paralinguistics, enhancing speech recognition accuracy and event localization capabilities.
Limitations & Outlook
The system experiences some lexical accuracy degradation in English speech recognition, requiring further optimization. Certain fine-grained NVV categories remain challenging to recognize, possibly needing more targeted supervision.
Plain Language Accessible to non-experts
Imagine you're in a kitchen, and traditional speech recognition systems are like chefs who only focus on the ingredients, ignoring all the sounds of cooking, like stirring, chopping, and boiling. This paper's system is like an all-around chef who not only focuses on the ingredients but also recognizes these sounds, helping to better understand the entire cooking process. In this way, the system can more accurately recognize nonverbal vocalizations in speech, such as laughter and sighs, enhancing dialogue understanding and media analysis capabilities.
ELI14 Explained like you're 14
Hey there! Imagine you're playing a super cool game with lots of characters talking. Traditional speech recognition systems are like players who only focus on what the characters say, ignoring sounds like laughter and sighs. This paper's system is like an all-around player who not only understands what the characters say but also recognizes these sounds, making the game experience more realistic! Isn't that awesome?
Glossary
NVV (Nonverbal Vocalizations)
Sounds that convey affective and interactional information in conversations, like laughter and sighs.
Used in this paper to enhance speech recognition accuracy.
ASR (Automatic Speech Recognition)
Technology that converts speech signals into text.
Used in this paper to recognize lexical and NVV content in bilingual settings.
Whisper-medium
A model used for speech recognition that supports inline tag generation.
Serves as the foundation for the NVV-aware ASR system in this paper.
Data Augmentation
Methods like adding noise and speed perturbation to increase data diversity.
Used to improve the diversity of public NVV corpora.
Multimodal LLM Filtering
Uses multimodal large language models to filter unreliable labels.
Enhances annotation consistency in corpora.
Open Questions Unanswered questions from this research
- 1 How to further improve NVV recognition accuracy without affecting lexical recognition?
- 2 How to optimize NVV recognition and annotation across different languages and cultural contexts?
Applications
Immediate Applications
Dialogue Understanding
Enhances the ability to recognize affective and interactional information in conversations, improving human-computer interaction.
Media Analysis
Identifies and analyzes nonverbal vocalizations in media content, improving content understanding and classification accuracy.
Long-term Vision
Multilingual Speech Recognition
Achieves broader multilingual support, enhancing global speech recognition capabilities.
Abstract
Nonverbal vocalizations (NVVs), such as laughter, sighs, breaths, and coughs, convey affective and interactional information that conventional automatic speech recognition (ASR) systems often discard. We present a bilingual Mandarin-English system for Track 1 of the NVVSpeech Challenge at ISCSLP 2026, which requires joint transcription of lexical content and 16 NVV categories at their transcript-relative positions. Our NVV-Aware Whisper adapts Whisper-medium through checkpoint-compatible vocabulary remapping, enabling lexical tokens and inline NVV tags to be decoded within a unified autoregressive sequence without expanding the vocabulary. To provide reliable and diverse supervision, we further introduce a source-adaptive data curation strategy that refines public NVV corpora through acoustic augmentation and multimodal LLM filtering, while mining spontaneous NVVs from in-the-wild media through automated preprocessing and annotation. Under the official bilingual evaluation protocol, the proposed system improves final score from 33.32 to 53.61, with ablations confirming the complementary benefits of the proposed data-curation components.