From Awareness to Adherence: Bridging the Context Gap in Spoken Dialogue Systems via Context-Aware Decoding
Proposed audio-adapted Context-Aware Decoding (CAD) method enhances context adherence in spoken dialogue systems.
Key Findings
Methodology
This study introduces an audio-adapted Context-Aware Decoding (CAD) method, leveraging internal attention mechanisms to identify key historical rounds and amplify multimodal contextual signals.
Key Results
- On the Audio MultiChallenge benchmark, the Semantic Memory task's average pass rate increased by 13.30%.
- In the Self Coherence task, the Qwen3-Omni model showed significant improvement.
- Ablation studies indicate that selecting the last four attention layers yields optimal results.
Significance
This method significantly improves context adherence in spoken dialogue systems during multi-turn conversations, addressing long-standing issues of inconsistent generation.
Technical Contribution
By precisely identifying and isolating key context, it resolves generation flaws and provides a new decoding intervention strategy.
Novelty
First to explicitly address generation failures not just as memory issues but as a gap between latent awareness and active adherence.
Limitations
- In some scenarios, it may fail to identify all relevant contexts, leading to performance drops.
- Extending to other modalities may require additional adjustments.
Future Work
Future work could explore more refined context selection mechanisms and applications in other modalities.
AI Executive Summary
Maintaining context consistency in multi-turn spoken dialogue systems has been a challenge. While existing models can recognize relevant historical information, they are often overshadowed by strong parametric priors during decoding. To address this, the study proposes an audio-adapted Context-Aware Decoding (CAD) method, leveraging internal attention mechanisms to identify key historical rounds and amplify multimodal contextual signals. Experimental results demonstrate significant improvements in Semantic Memory and Self Coherence tasks on the Audio MultiChallenge benchmark. This study not only resolves generation flaws but also provides a new decoding intervention strategy for spoken dialogue systems with broad application prospects. Future work will explore more refined context selection mechanisms and applications in other modalities.
Deep Analysis
Background
In recent years, end-to-end spoken dialogue systems have made significant progress in handling complex multi-turn conversations. However, maintaining dialogue context consistency remains a challenge, especially in long conversations.
Core Problem
Existing models often forget dialogue history in multi-turn conversations, leading to responses that are inconsistent with the context, affecting user experience.
Innovation
Proposed audio-adapted Context-Aware Decoding method, leveraging internal attention mechanisms to identify key historical rounds and amplify multimodal contextual signals.
Methodology
- �� Use attention mechanisms to identify key historical rounds
- �� Contrast output distributions with and without key context
- �� Directly amplify multimodal contextual signals
Experiments
Evaluated on the Audio MultiChallenge benchmark, focusing on Semantic Memory and Self Coherence tasks, using gpt-5-nano as the evaluation model.
Results
In the Semantic Memory task, the Qwen3-Omni model's average pass rate increased from 22.67% to 39.33%.
Applications
The method can be used to enhance context adherence in voice assistants during multi-turn conversations, improving user experience.
Limitations & Outlook
In some scenarios, it may fail to identify all relevant contexts, leading to performance drops. Extending to other modalities may require additional adjustments.
Plain Language Accessible to non-experts
Imagine a kitchen where the chef needs to remember the ingredients and steps for each dish. Existing chefs can remember some key steps but are often distracted by other factors, leading to dishes that don't meet expectations. Our method is like giving the chef a new tool to help him focus on key steps while cooking, ensuring each dish meets the customer's requirements.
ELI14 Explained like you're 14
Imagine playing a complex game where the character needs to remember previous tasks and clues. Existing game characters can remember some clues but are often distracted by other factors, leading to task failures. Our method is like giving the game character a new tool to help him focus on key clues while executing tasks, ensuring successful task completion.
Glossary
Context-Aware Decoding
A method to improve generation quality by identifying and enhancing key historical information.
Used to enhance context adherence in spoken dialogue systems.
Attention Mechanism
A technique used to identify and focus on important information.
Used to identify key historical rounds.
Multimodal Signals
Signals combining multiple sources of information, such as audio and text.
Used to enhance contextual information.
Semantic Memory
The model's memory of user constraints when generating responses.
Evaluates the model's context adherence.
Self Coherence
The model's consistency with its own prior generations when generating responses.
Evaluates the model's context adherence.
Open Questions Unanswered questions from this research
- 1 How to implement similar context-aware decoding in other modalities?
- 2 How to further optimize context selection mechanisms to improve performance?
Applications
Immediate Applications
Voice Assistants
Enhance context adherence in voice assistants during multi-turn conversations, improving user experience.
Long-term Vision
Multimodal Dialogue Systems
Extend to other modalities to improve dialogue systems' performance in complex scenarios.
Abstract
Despite the success of end-to-end (E2E) spoken dialogue systems, maintaining strict context adherence in multi-round conversations remains a challenge. While prior works attribute these failures to models forgetting dialogue history, we highlight an equally critical but overlooked bottleneck: a gap between latent context awareness and active adherence. Although models internally recognize relevant past utterances, strong parametric priors often overshadow these signals during decoding. To bridge this gap, we propose an audio-adapted Context-Aware Decoding (CAD) approach. By leveraging internal attention mechanisms to isolate key historical rounds, our approach contrasts output distributions with and without this key context during inference, directly amplifying multimodal contextual signals. Evaluations on the Audio MultiChallenge benchmark demonstrate significant improvements in Semantic Memory and Self Coherence subtasks, successfully enforcing strict, context-faithful adherence.