Preference Optimization with LALM Feedback for Continuous Autoregressive Non-Verbal Vocalization Generation
Proposed a preference optimization framework with LALM feedback for NVV generation, achieving a final score of 75.80.
Key Findings
Methodology
The paper introduces a preference optimization framework using Large Audio-Language Model (LALM) feedback for non-verbal vocalization (NVV) generation. A bilingual prompt corpus is constructed, followed by stochastic model rollouts and LALM evaluation to create preference data pairs. A two-stage optimization strategy is employed: Rejection Sampling Fine-Tuning (RSFT) adapts to high-scoring samples, and Anchored Flow-DPO optimizes preferences using flow-matching loss.
Key Results
- On the NVVSpeech Challenge Track 2 test set, the method achieves a Final Track2Score of 75.80 (79.39 ZH / 72.21 EN), outperforming the VoxCPM2 baseline by +1.84.
- Improvements are mainly observed in NVV Accuracy and NVV Perceptual Effect, while Overall Quality remains stable.
- Experiments demonstrate that combining RSFT and Flow-DPO significantly enhances preference learning.
Significance
This study significantly improves the accuracy and perceptual quality of NVV generation through a preference optimization framework, addressing the data scarcity issue in traditional supervised learning, providing new insights and methods for the speech generation field.
Technical Contribution
Technical contributions include: introducing a DPO-style preference learning method without explicit sequence likelihoods, using flow-matching loss for preference optimization, and automating preference data construction with LALM feedback.
Novelty
This is the first application of preference optimization to continuous autoregressive speech generation, proposing a novel method using flow-matching loss as a preference signal, differing from traditional sequence likelihood methods.
Limitations
- The model may struggle with complex contexts in NVV generation, requiring further optimization.
- Preference data construction relies on LALM scoring, which may introduce scoring bias.
Future Work
Future work could explore preference optimization methods in more complex contexts and further refine LALM scoring mechanisms to enhance preference data accuracy.
AI Executive Summary
Non-verbal vocalizations (NVVs) like laughter and sighs play a crucial role in conveying emotions and intentions. However, existing speech generation systems face challenges in controlling NVVs, especially in maintaining naturalness and overall acoustic quality while accurately realizing NVV types and occurrences.
This paper proposes a preference optimization framework using Large Audio-Language Model (LALM) feedback for NVV generation. By constructing a bilingual prompt corpus and automating preference data pairs, a two-stage optimization strategy is employed: Rejection Sampling Fine-Tuning (RSFT) adapts to high-scoring samples, and Anchored Flow-DPO optimizes preferences using flow-matching loss.
Experimental results show that on the NVVSpeech Challenge Track 2 test set, the method achieves a final score of 75.80, outperforming the VoxCPM2 baseline by +1.84, with improvements mainly in NVV Accuracy and NVV Perceptual Effect while maintaining stable overall quality. This study provides new insights and methods for the speech generation field, addressing the data scarcity issue in traditional supervised learning.
Deep Dive
Abstract
We propose a preference optimization framework with Large Audio-Language Model (LALM) feedback for controllable non-verbal vocalization (NVV) generation in continuous autoregressive speech models. To construct preference data without human preference annotation, we build a bilingual prompt corpus by combining NVV-injected real transcripts with LLM-generated semantically aligned prompts, perform stochastic model rollouts, and use a LALM to rank candidate utterances and form same-prompt chosen--rejected pairs. We then adopt a two-stage optimization strategy: Rejection Sampling Fine-Tuning (RSFT) first adapts the model to LALM-selected high-scoring samples, followed by Anchored Flow-DPO, which formulates pairwise preference optimization using utterance-level flow-matching loss and retains the chosen-sample flow-matching objective as an SFT anchor. This design enables DPO-style preference learning without explicit sequence likelihoods while preserving direct supervision on preferred realizations. On the official 1,600-utterance NVVSpeech Challenge Track~2 test set, our method achieves a Final Track2Score of \textbf{75.80} (79.39 ZH / 72.21 EN), outperforming the VoxCPM2 baseline by \textbf{+1.84}. The improvements are mainly driven by higher NVV Accuracy and NVV Perceptual Effect, while Overall Quality remains stable.