Multimodal Emotion Recognition in Conversations via Class-Wise Adaptive Modality Fusion and Affective Geometry
An SDT extension combining geometric vision, class-wise fusion, and affective priors reaches 75.93% and 74.11% weighted F1 on MELD and IEMOCAP.
Key Findings
Methodology
The paper extends Self-Distillation Transformer (SDT). Text uses RoBERTa-large, audio uses emotion2vec, and vision combines ViT appearance features with 3DDFA-V2 landmarks, expression parameters, and FACS action units. Intra- and inter-modal Transformers encode context; a class-wise MLP estimates modality relevance for every emotion and fuses unimodal logits. A Russell valence-arousal prior then regularizes emotionally implausible transitions.
Key Results
- Adding appearance-plus-geometry vision to the updated SDT baseline raises weighted F1 from 75.49 to 75.76 on MELD and from 69.50 to 73.86 on IEMOCAP. Relative to appearance-only visual features, the abstract reports gains of 0.27 and 4.36 points, respectively.
- Class-wise adaptive fusion further reaches 75.93% and 74.11% weighted F1 on MELD and IEMOCAP, improving over the original softmax gate by 0.17 and 0.25 points. IEMOCAP accuracy reaches 74.72%.
- The valence-arousal prior targets shifted utterances: accuracy improves by 0.30 points on MELD and 0.74 on IEMOCAP while stable-turn performance is preserved, indicating selective rather than globally biased correction.
Significance
The study addresses three persistent ERC problems: textual dominance in multimodal fusion, emotion-dependent modality reliability, and the lack of structure in discrete emotion labels. Its results suggest that structured facial measurements complement rather than replace holistic appearance features. Class-conditioned reliability estimation also better matches the fact that vocal energy, lexical content, and facial configuration are informative for different emotions. The approach is relevant to meeting analytics, affect-aware interfaces, and assistive communication, although deployment requires careful privacy and robustness controls.
Technical Contribution
The model retains SDT’s intra-modal Transformers, inter-modal Transformers, and objective L=γ1LTask+γ2LCE+γ3LKL, but replaces shared softmax representation gating with class-conditioned logit fusion. For each modality, an MLP receives the enhanced representation and detached unimodal probabilities, producing class scores that are normalized across modalities. Vision symmetrically adds normalized ViT and geometry branches. The affective prior computes expected valence-arousal, transition magnitude, and candidate distances, then adds a temperature-scaled log-prior to fused logits.
Novelty
The novelty is the integration of three complementary mechanisms in one SDT-style ERC pipeline rather than the isolated invention of any single component. Compared with DialogueRNN, MMGCN, and original SDT, the work jointly introduces structured facial evidence, emotion-class-dependent modality selection, and a transition-aware posterior correction. The contribution is therefore architectural and functional: it links evidence reliability with affective structure.
Limitations
- The training subsets of the released emotion2vec checkpoint are undisclosed, so possible overlap with MELD or IEMOCAP cannot be ruled out. The checkpoint is not fine-tuned, making leakage and domain adaptation difficult to assess conclusively.
- The visual stream depends on face detection, active-speaker detection, and 16-frame sampling. Occlusion, profile views, crowding, or missing video produce zero embeddings; geometry is also sensitive to pose and landmark noise.
- IEMOCAP uses Session 5 for both validation and testing, and the datasets have different label systems, limiting claims about generalization.
Future Work
Future research should learn or personalize valence-arousal coordinates, model predictive uncertainty, and train robustly with missing modalities. Evaluation should expand across languages, cultures, recording conditions, speakers, and strict cross-dataset splits. Transparent deduplication and domain adaptation of pretrained models such as emotion2vec would improve benchmark validity. Longer-context affect modeling could also replace the current adjacent-turn prior with a hierarchical trajectory model.
AI Executive Summary
Emotion recognition in conversation requires more than reading words. Meaning is shaped by vocal delivery, facial behavior, speaker context, and the way affect changes from one turn to the next. Yet multimodal systems often let text dominate, use one shared gate for every emotion, and treat labels such as happy, excited, and sad as unrelated categories. These design choices make shifted or ambiguous utterances especially difficult.
Oriol Marín and colleagues extend the Self-Distillation Transformer with three coordinated mechanisms. The visual stream combines ViT appearance embeddings with 3DDFA-V2 landmarks, expression parameters, and FACS action units. A class-wise adaptive fusion module uses a lightweight MLP to estimate how informative text, audio, and vision are for each emotion, then combines their logits. Finally, a Russell valence-arousal prior discourages implausibly large affective jumps between neighboring utterances.
On MELD and IEMOCAP, appearance-plus-geometry vision improves weighted F1 over appearance-only features by 0.27 and 4.36 points. Class-wise fusion adds another 0.17 and 0.25 points over the original softmax gate, producing 75.93% and 74.11% weighted F1. The affective prior raises accuracy on shifted utterances by 0.30 and 0.74 points without harming stable turns. The broader lesson is that structured facial evidence, emotion-dependent modality weighting, and affective geometry provide complementary—not competing—routes to more reliable multimodal ERC.
Deep Analysis
Background
ERC models emotion at utterance level while tracking context, speakers, and temporal dynamics. CMN, ICON, DialogueRNN, and DialogueGCN introduced memories, speaker states, and relational graphs; HiTrans, DialogueTRM, MM-DFN, and SDT brought Transformer-based contextual and cross-modal modeling. However, common fusion mechanisms rarely estimate modality reliability explicitly, and independent class labels ignore distances between emotions.
Core Problem
A modality’s value depends on the target emotion and signal quality: audio may reveal high arousal, while facial structure may distinguish expression-specific classes. Video can also be corrupted by occlusion, lighting, pose, or multiple people. Separately, a classifier that treats classes independently has no principled reason to prefer happy-to-excited over happy-to-sad, making emotional shifts vulnerable to implausible predictions.
Innovation
- �� Appearance-plus-geometry vision combines holistic ViT features with landmarks, expression parameters, and action units. • Class-wise adaptive fusion predicts a separate text/audio/vision contribution for each class instead of sharing one gate. • The valence-arousal prior represents emotions in Russell’s continuous affective plane and biases predictions toward plausible transitions. Together, the components improve evidence quality, selection, and temporal consistency.
Methodology
- �� Text: RoBERTa-large encodes the target utterance preceded by two dialogue turns and speaker identifiers. • Audio: emotion2vec processes mono, 16-kHz audio truncated to 10 seconds. • Vision: MTCNN detects faces, LightASD selects the active speaker, and 16 frames are sampled per utterance; ViT yields 768-dimensional appearance features, while geometry produces 408-dimensional landmarks, 39-dimensional expression/pose statistics, and 60-dimensional action-unit statistics. • All streams are projected to d=1024 and processed by intra-/inter-modal Transformers. • MLPs produce rcmi, modality softmax yields wcmi, and fused logits are ẑci=Σmwcmi zcmi. • The prior adds αβiϕi based on valence-arousal distances.
Experiments
MELD contains 13,708 utterances from 1,433 multiparty dialogues and seven classes, with official 9,989/1,109/2,610 train/validation/test splits. IEMOCAP contains 7,433 utterances, six classes, and uses Sessions 1–4 for training and Session 5 for validation/testing. Weighted F1 is primary, with accuracy as a complement. MELD uses learning rate 5×10^-6, batch size 8, and distillation temperature 8; IEMOCAP uses 10^-4, 16, and 1. Ablations add geometry, adaptive fusion, and the affective prior progressively.
Results
The updated SDT baseline obtains 75.49/69.50 weighted F1 on MELD/IEMOCAP. Appearance-plus-geometry vision raises this to 75.76/73.86, and class-wise fusion reaches 75.93/74.11; the cross-dataset mean rises from 72.50 to 75.02. IEMOCAP benefits especially strongly from geometry. Original SDT reference scores are 66.60/74.08 weighted F1 but were not reproduced under the new feature setting. The prior adds 0.30/0.74 accuracy points on shifted turns.
Applications
Potential uses include meeting emotion analytics, customer-service monitoring, online learning feedback, and affect-aware robots. Deployment requires reliable transcription, speaker localization, face processing, consent, and privacy protection. The class-wise mechanism can also support explanations by reporting whether a prediction relied primarily on lexical, vocal, or facial evidence. Missing-modality handling should be monitored rather than silently interpreted as neutral evidence.
Limitations & Outlook
The pipeline is computationally substantial, combining face processing, ViT, multiple Transformers, and three modalities. The affective prior depends on the choice of canonical or dataset-specific coordinates, and it only examines adjacent turns rather than long-range emotional trajectories. Data diversity is limited, labels differ across datasets, and IEMOCAP’s validation/testing protocol complicates reproducibility. Future work should test strict cross-speaker and cross-domain generalization, uncertainty calibration, longer context, and robust missing-modality training.
Plain Language Accessible to non-experts
Imagine a service team trying to infer a customer’s mood. One worker reads the customer’s words, another listens to pitch and speaking speed, and a third watches facial movement. A conventional team might give all three a single shared voting rule. This paper gives them a smarter system: when deciding whether the customer is angry, one worker may matter most; when deciding whether the customer is sad, another may be more trustworthy.
The face specialist also does more than inspect the whole face like a photograph. It measures how eyebrows, eyes, mouth, and head position move, much like making a map of facial actions. The team then checks whether the new mood is a believable continuation. If the previous turn was happy, excited or pleased may be more plausible than a sudden jump to sadness—but the check is only a nudge, not a ban.
Tests on MELD and IEMOCAP show that combining words, voice, appearance, and facial movement works better than relying on one view. The main message is simple: understanding emotion requires both the evidence itself and a sensible idea of how feelings can change. The system still struggles when video is blurry, faces are hidden, or people express feelings indirectly.
ELI14 Explained like you're 14
Suppose you are watching a group chat video and must guess the feeling behind every message. Subtitles alone can fool you: “Great job!” might be genuine praise or sarcasm. So the model hires three detectives. The word detective reads the sentence, the voice detective listens to tone and energy, and the face detective watches expressions.
This face detective is extra careful. Instead of only asking whether the whole face looks happy, it tracks eyebrow, eye, mouth, and head movements—like drawing a movement map. The model also changes its teamwork rules depending on the emotion. For excitement, voice might be a strong clue; for a clearly visible expression, the face might matter more. Why use one voting rule for every emotion?
Then comes a reality check. If someone was happy one moment and excited the next, that is not surprising. If they suddenly become very sad, the model asks for stronger evidence. It does not forbid surprises; it simply gives nearby feelings a small advantage when the evidence is weak.
The experiments show real gains on MELD and IEMOCAP. Facial structure helps, emotion-specific teamwork helps again, and the transition check improves changing turns. But blurry video, blocked faces, sarcasm, and cultural differences can still confuse the detectives. So this is a useful assistant—not a magical mind reader!
Glossary
Emotion Recognition in Conversations (对话情绪识别)
The task of assigning an emotion to each utterance in a dialogue. Unlike isolated classification, it uses context, speakers, and affective dynamics.
The paper evaluates multimodal ERC on MELD and IEMOCAP.
Self-Distillation Transformer (自蒸馏Transformer)
An architecture that encodes multiple modalities while training unimodal branches to imitate the fused prediction. KL divergence transfers information from the multimodal teacher.
The proposed model preserves SDT’s intra-modal, inter-modal, and distillation components.
Class-wise adaptive fusion（类别自适应融合）
A fusion strategy that estimates a separate modality contribution for every emotion class. It is more granular than one shared gate across all classes.
The paper uses MLP scores rcmi and modality-wise softmax weights wcmi.
Affective geometry（情绪几何）
A continuous representation in which emotional categories have positions and distances. Here the dimensions are valence and arousal.
The geometry supplies a prior for emotion transitions.
Facial Action Units（面部动作单元）
Interpretable facial muscle actions defined by FACS, such as eyebrow or mouth movements. They describe expression dynamics rather than identity alone.
Twenty action units are summarized by means, standard deviations, and frame differences.
Weighted F1（加权F1）
A class-sensitive F1 average weighted by class frequency. It is useful when emotion categories are imbalanced.
It is the primary evaluation metric for both datasets.
Open Questions Unanswered questions from this research
- 1 Should valence-arousal coordinates be universal or dataset-specific? Cultural conventions and annotation styles may change affective distances, so stable cross-dataset coordinate learning remains unresolved.
- 2 How should models represent long conversations containing sarcasm, repeated reversals, and implicit emotion? The current prior only compares adjacent turns and cannot explain long-range trajectories.
- 3 Because emotion2vec training subsets are undisclosed, rigorous deduplication, privacy auditing, and fair comparison remain open methodological requirements for multimodal ERC.
Applications
Immediate Applications
Customer-service and meeting analytics
A platform can combine transcripts, call audio, and optional video to flag anger, sadness, or satisfaction. It needs speaker separation, reliable face processing, consent, and privacy controls. Outputs should include emotion changes and the dominant evidence source rather than an unexplained label.
Online learning feedback
Educational systems could estimate confusion, engagement, or frustration from student speech, words, and optional facial cues. Camera opt-out is essential, and results should guide aggregate teaching feedback rather than impose high-stakes judgments on individuals.
Long-term Vision
Trustworthy affect-aware agents
Future robots and virtual assistants could combine multimodal evidence with affective geometry to interpret users without overreacting to brief tone changes. Progress requires cross-cultural data, interpretable uncertainty, real-time efficiency, and strong ethical governance.
Abstract
Emotion Recognition in Conversations (ERC) requires integrating heterogeneous textual, audio, and visual cues while accounting for conversational context and emotional dynamics. We extend the Self-Distillation Transformer architecture for ERC with appearance+geometry visual representations, class-wise adaptive modality fusion, and a valence-arousal prior for affective transitions. On the MELD and IEMOCAP datasets, geometry-enhanced visual representations improve weighted F1 by 0.27 and 4.36 points over appearance-only features, respectively, while class-wise adaptive fusion provides further gains of 0.17 and 0.25 points over the original softmax gate. The valence-arousal prior yields targeted improvements of 0.30 and 0.74 accuracy points on emotionally shifted utterances while preserving performance on stable turns. These results indicate that structured facial cues, emotion-dependent modality weighting, and affective geometry provide complementary benefits for multimodal ERC.