MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations
MELD extends EmotionLines with multimodal, multi-party emotion recognition, offering 13,000 annotated utterances.
Key Findings
Methodology
MELD integrates text, audio, and visual modalities, annotating 13,000 utterances from 1,433 dialogues in Friends. Baseline models include bcLSTM and DialogueRNN, which uses GRUs to model multi-party emotional dynamics.
Key Results
- Result 1: DialogueRNN achieved the best F1 score of 60.25% for emotion classification, outperforming text-only models (57.03%).
- Result 2: Multimodal fusion improved accuracy for short utterances (<5 words) and complex emotions like 'surprise'.
- Result 3: Re-annotation improved Fleiss' kappa from 0.34 to 0.43, demonstrating the value of multimodal information.
Significance
MELD fills the gap in multimodal, multi-party emotion recognition datasets, enabling more realistic conversational emotion modeling. It advances dialogue systems and human-computer interaction research.
Technical Contribution
Contributions include: 1) the first large-scale multimodal multi-party emotion dataset; 2) DialogueRNN for modeling multi-party emotional dynamics; 3) strong multimodal baselines validating the benefits of fusion.
Novelty
MELD is the first dataset combining text, audio, and visual modalities for multi-party dialogues, surpassing IEMOCAP and SEMAINE in scale and complexity.
Limitations
- Limitation 1: Data is sourced from Friends, which may introduce cultural and contextual biases.
- Limitation 2: Imbalanced emotion distribution, with fewer samples for 'disgust' and 'fear'.
- Limitation 3: Visual feature extraction is not fully optimized, potentially limiting performance.
Future Work
Future directions include expanding the dataset to diverse contexts, improving visual feature extraction, and exploring advanced multimodal fusion techniques.
AI Executive Summary
The MELD dataset is a groundbreaking resource for multimodal, multi-party emotion recognition, extending EmotionLines with 13,000 utterances across text, audio, and visual modalities. Baseline models like bcLSTM and DialogueRNN leverage GRUs to model emotional dynamics in multi-party conversations.
Experiments demonstrate that multimodal fusion significantly improves emotion recognition accuracy, especially for short utterances and complex emotions. Re-annotation also improved data consistency, validating the importance of multimodal information.
While MELD represents a significant advancement in scale and diversity, it faces challenges such as data imbalance and cultural bias. Future work will focus on expanding the dataset and optimizing multimodal feature extraction to further advance conversational emotion recognition research.
Deep Analysis
Background
Emotion recognition is crucial for dialogue systems and human-computer interaction. Existing datasets like IEMOCAP and SEMAINE are limited in scale and support only dyadic conversations. EmotionLines introduced multi-party dialogues but lacked multimodal data.
Core Problem
The lack of large-scale multimodal, multi-party emotion datasets limits research on complex emotional dynamics. Existing datasets fail to capture multimodal cues like vocal tone and facial expressions.
Innovation
Key innovations include: 1) integrating text, audio, and visual modalities; 2) providing strong baselines like DialogueRNN for modeling emotional dynamics; 3) re-annotating data for improved consistency.
Methodology
- �� Dataset construction: Extracted dialogues from EmotionLines, adding audio and visual modalities.
- �� Annotation: Re-annotated emotions using multimodal cues with majority voting.
- �� Baseline models: Included bcLSTM and DialogueRNN, the latter using GRUs to model multi-party emotional dynamics.
Experiments
Experiments used the MELD dataset with baselines like text-CNN, bcLSTM, and DialogueRNN. Metrics included F1 scores, and ablation studies were conducted to evaluate multimodal fusion.
Results
DialogueRNN achieved the highest F1 score of 60.25%. Multimodal fusion significantly improved accuracy for short utterances and complex emotions. Re-annotation enhanced data consistency.
Applications
MELD enables the development of multimodal emotion recognition systems for customer service bots, mental health monitoring, and education.
Limitations & Outlook
Limitations include single-source data, imbalanced emotion distribution, and suboptimal visual feature extraction. Future work can address these through dataset expansion and model optimization.
Plain Language Accessible to non-experts
Imagine a detective solving a case. They don’t just listen to what witnesses say (text); they also observe their tone (audio) and facial expressions (visual). MELD is like a training kit for AI detectives, helping them interpret emotions in conversations more effectively.
ELI14 Explained like you're 14
Think about chatting with friends. Sometimes you can tell how they feel just by their tone or expression, right? MELD teaches AI to do the same! It’s like giving a robot the ability to read emotions from voices and faces, not just words.
Glossary
Multimodal
Combining multiple information sources like text, audio, and visuals.
MELD integrates multimodal data to improve emotion recognition.
Emotion Recognition
Identifying emotions from signals like text, speech, or visuals.
The study uses MELD for emotion recognition experiments.
GRU (Gated Recurrent Unit)
A lightweight recurrent neural network for sequence modeling.
DialogueRNN uses GRUs to model emotional dynamics.
Fleiss' kappa
A statistical measure of inter-annotator agreement.
Re-annotation improved Fleiss' kappa from 0.34 to 0.43.
Ablation Study
Evaluating the contribution of components by removing them.
Ablation studies validated the role of multimodal fusion.
Open Questions Unanswered questions from this research
- 1 How can multimodal information be fused more efficiently?
- 2 How can datasets be expanded to cover diverse cultures and contexts?
Applications
Immediate Applications
Customer Service Bots
Develop smarter bots that understand user emotions for personalized responses.
Mental Health Monitoring
Use emotion recognition to track emotional changes for mental health interventions.
Long-term Vision
Multimodal Human-AI Interaction
Enable natural interactions by enhancing AI’s emotional understanding.
Abstract
Emotion recognition in conversations is a challenging task that has recently gained popularity due to its potential applications. Until now, however, a large-scale multimodal multi-party emotional conversational database containing more than two speakers per dialogue was missing. Thus, we propose the Multimodal EmotionLines Dataset (MELD), an extension and enhancement of EmotionLines. MELD contains about 13,000 utterances from 1,433 dialogues from the TV-series Friends. Each utterance is annotated with emotion and sentiment labels, and encompasses audio, visual and textual modalities. We propose several strong multimodal baselines and show the importance of contextual and multimodal information for emotion recognition in conversations. The full dataset is available for use at http:// affective-meld.github.io.