Dual-Scale State-Space Modeling with Speaker-Wise Dynamic CRF for Speech Emotion Recognition in Conversation
DSSM-CRF achieves 75.81% UA on IEMOCAP, addressing cross-speaker influence in conversational emotion recognition.
Key Findings
Methodology
DSSM-CRF employs dual-scale state-space models and speaker-wise dynamic CRF to encode self-supervised speech representations at frame and dialogue levels. Each speaker's utterances form independent CRF chains, combining a global transition matrix with contextual prediction residuals. An auxiliary objective supervises emotion shifts but does not participate in Viterbi inference.
Key Results
- On IEMOCAP, DSSM-CRF achieved 75.81% UA and 74.90% WA, significantly outperforming existing methods.
- On MELD, DSSM-CRF achieved 54.72% WA and 49.31% WF1, demonstrating adaptability to multi-party conversations.
- Ablation studies show that removing contextual transition residuals has the greatest impact, highlighting its importance in emotion recognition.
Significance
This research addresses the long-standing issue of cross-speaker interference in emotion recognition by distinguishing dialogue-wide contextual influence from speaker-specific emotion evolution. It not only improves recognition accuracy but also provides new insights for multi-party conversational emotion analysis.
Technical Contribution
DSSM-CRF introduces dual-scale state-space models and speaker-wise dynamic CRF, differing from existing methods' single dialogue-level CRF chains. Its innovation lies in combining a global transition matrix with contextual prediction residuals, offering new engineering possibilities.
Novelty
DSSM-CRF is the first to explicitly separate dialogue context from speaker emotion trajectories in emotion recognition, providing a novel approach to modeling emotion shifts compared to existing methods.
Limitations
- There is room for improvement in recognizing emotion shifts, especially in accurately capturing speaker emotion changes.
- Efficiency in handling long conversations needs enhancement.
Future Work
Future work could explore more efficient methods for modeling emotion shifts and validate on larger datasets.
AI Executive Summary
Conversational emotion recognition requires balancing dialogue context and speaker emotion evolution. Existing methods often conflate cross-speaker influence with single-speaker emotion trajectories, leading to suboptimal accuracy. DSSM-CRF distinguishes these structural roles using dual-scale state-space models and speaker-wise dynamic CRF. Experimental results show superior performance on IEMOCAP and MELD datasets, significantly enhancing emotion recognition accuracy.
The core technology of DSSM-CRF lies in using bidirectional state-space encoders to encode self-supervised speech representations at frame and dialogue levels. Each speaker's utterances form independent CRF chains, combining a global transition matrix with contextual prediction residuals. This design allows interlocutor turns to influence contextual emotion scores without being treated as transitions in another speaker's emotion trajectory.
Despite DSSM-CRF's excellent performance, there is room for improvement in recognizing emotion shifts. Future research could explore more efficient methods for modeling emotion shifts and validate on larger datasets to further enhance conversational emotion recognition performance.
Deep Analysis
Background
The field of emotion recognition has evolved from handcrafted feature extraction to self-supervised learning models. Early methods relied on manually designed acoustic features and traditional classifiers, while recent research leverages self-supervised learning models to extract transferable representations directly from raw waveforms. Conversational emotion recognition differs from isolated utterance classification because emotion depends not only on its acoustic realization but also on surrounding interactions and the speaker's emotional history.
Core Problem
The core problem in conversational emotion recognition is how to simultaneously model dialogue context and speaker emotion evolution. Existing methods often conflate cross-speaker responses with single-speaker transitions, failing to adapt to individual utterance pairs' emotion changes. Solving this problem is crucial for improving emotion recognition accuracy.
Innovation
DSSM-CRF's core innovations include: 1) using dual-scale state-space models to encode self-supervised speech representations at frame and dialogue levels, 2) introducing speaker-wise dynamic CRF to organize each speaker's utterances into independent CRF chains, 3) combining a global transition matrix with contextual prediction residuals to enhance emotion shift modeling flexibility.
Methodology
- �� Use WavLM-Large to extract frame-level representations and generate utterance representations through multi-scale attentive statistics pooling.
- �� At the frame level, process frame sequences using bidirectional Mamba-2-style state-space encoders.
- �� At the dialogue level, augment each utterance representation and generate contextual representations using bidirectional state-space models.
- �� Organize each speaker's utterances into independent CRF chains, combining a global transition matrix with contextual prediction residuals for emotion shift modeling.
Experiments
Experiments were conducted on IEMOCAP and MELD datasets, using four-class and seven-class emotion settings, respectively. IEMOCAP was evaluated with five-fold cross-validation, while MELD used the official data split. Reported metrics include unweighted accuracy (UA), weighted accuracy (WA), and weighted F1-score (WF1). Ablation studies assessed the contribution of each component.
Results
On IEMOCAP, DSSM-CRF achieved 75.81% UA and 74.90% WA, significantly outperforming existing methods. On MELD, DSSM-CRF achieved 54.72% WA and 49.31% WF1, demonstrating adaptability to multi-party conversations. Ablation studies show that removing contextual transition residuals has the greatest impact, highlighting its importance in emotion recognition.
Applications
DSSM-CRF can be applied in emotion-aware assistants, call center analytics, and behavioral monitoring. Its improvements in multi-party conversational emotion recognition enhance the accuracy and user experience of these applications.
Limitations & Outlook
Despite DSSM-CRF's excellent performance, there is room for improvement in recognizing emotion shifts. Efficiency in handling long conversations needs enhancement. Future research could explore more efficient methods for modeling emotion shifts and validate on larger datasets.
Plain Language Accessible to non-experts
Imagine a factory where workers are on different production lines. Each worker represents a speaker, and their work status represents emotion. DSSM-CRF is like an intelligent management system that can monitor each worker's status and the entire factory's production. By analyzing each worker's work record (frame-level representation) and the entire production line's operation (dialogue-level representation), it predicts workers' emotion changes. This system not only recognizes current worker status but also predicts their emotion change trends, helping managers better allocate tasks.
ELI14 Explained like you're 14
Imagine you're playing an online multiplayer game, and each player has their own emotion status. DSSM-CRF is like a super AI assistant in the game that analyzes each player's emotion changes, helping you better understand your teammates' moods. This AI assistant not only recognizes players' current emotions but also predicts their emotion change trends, making it easier to collaborate in the game. Isn't that cool?
Glossary
Conditional Random Field
A probabilistic model for sequence labeling that captures dependencies among labels.
Used in the paper to model speaker emotion trajectory transitions.
State-Space Model
A mathematical model for dynamic systems that describes state changes over time.
Used to encode frame-level and dialogue-level information of self-supervised speech representations.
Self-Supervised Learning
A machine learning method that trains on automatically generated labels from unlabeled data.
The foundational technique for extracting speech representations.
Viterbi Algorithm
A dynamic programming algorithm for finding the most likely sequence of states.
Used in CRF for decoding emotion trajectories.
Multi-Scale Attentive Statistics Pooling
A pooling method that combines multi-scale information to enrich feature representations.
A key step in generating utterance representations.
Open Questions Unanswered questions from this research
- 1 How to validate DSSM-CRF's effectiveness on larger datasets, especially its adaptability to different languages and cultural contexts.
- 2 How to improve DSSM-CRF's efficiency in handling long conversations, particularly in real-time applications.
Applications
Immediate Applications
Emotion-Aware Assistants
DSSM-CRF can be used to develop smarter emotion-aware assistants, helping users better understand emotional changes in conversations.
Long-term Vision
Multilingual Emotion Analysis
DSSM-CRF's framework can be extended to multilingual emotion analysis, aiding emotional understanding in cross-cultural communication.
Abstract
Conversational speech emotion recognition must reconcile acoustic evidence across temporal scales with two interaction processes: cross-speaker contextual influence and within-speaker emotion evolution. We propose DSSM-CRF, an audio-only architecture that explicitly separates these processes. Bidirectional state-space models encode fused self-supervised speech representations at frame and dialogue scales, so each utterance representation captures local prosody and context from all speakers. The decoder then orders each speaker's utterances into an independent dynamic conditional random field chain. Consecutive utterances in a speaker's chain form a transition pair whose score combines a corpus-level transition matrix with a residual predicted from the two contextualized utterances. An auxiliary objective supervises whether each pair changes emotion but does not participate in Viterbi inference. Thus, interlocutor turns affect contextual emotion scores without being treated as transitions in another speaker's emotion trajectory. DSSM-CRF achieves 75.81% UA and 74.90% WA on IEMOCAP, and 54.72% WA and 49.31% WF1 on MELD. Matched controls demonstrate complementary gains from speaker-wise factorization and CRF modeling.