EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models
EQ-Bench assesses LLMs' emotional understanding via dialogue-based emotion intensity ratings, achieving a correlation of 0.97 with broad intelligence benchmarks.
Key Findings
Methodology
This study develops a dialogue-based emotion intensity rating benchmark, where GPT-4 generates diverse conflict scenarios. Models rate characters’ emotions on a 0-10 scale, with scores normalized and differences computed to objectively assess emotional understanding. The process involves automated scoring, including self-critique and revision steps, ensuring consistency. The benchmark uses 60 questions, with results strongly correlating (r=0.97) with multi-domain benchmarks like MMLU, validating its effectiveness in measuring broad intelligence through emotional comprehension.
Key Results
- GPT-4 achieved an average score of 62.52, outperforming other models, indicating superior emotional understanding. Open-source models like SynthIA-70B follow closely, showing that fine-tuning for role-play enhances EI. Scores correlate highly with MMLU (r=0.97), HellaSwag (r=0.91), confirming the benchmark’s representativeness. Incorporating self-critique improved scores by 9.3%, highlighting the importance of reasoning and correction. The automated pipeline ensures reproducibility and fairness across models.
- Model performance varies significantly across versions, with newer GPT-4 versions scoring higher. The design of complex scenarios and nuanced scoring effectively distinguishes capabilities, demonstrating robustness and sensitivity.
- The strong correlation with broad intelligence benchmarks suggests emotional understanding as a proxy for general cognition. The benchmark’s automation and objectivity make it suitable for large-scale, fair evaluation, advancing AI’s social and emotional competence.
Significance
This work addresses the gap in standardized, objective evaluation of emotional intelligence in large language models. By correlating strongly with established multi-domain benchmarks, EQ-Bench demonstrates that emotional understanding is integral to broad intelligence. Its scalable, automated design facilitates consistent assessment across diverse models, supporting development of socially adept AI systems. The high correlation with perceived intelligence metrics underscores its practical relevance, promising improvements in human-AI interaction, social robotics, and mental health applications. It offers a new paradigm for AI evaluation, emphasizing nuanced social cognition.
Technical Contribution
The paper introduces a novel dialogue-based emotion rating framework, combining GPT-4 generated scenarios with a non-sum constraint scoring mechanism. The inclusion of self-critique and revision steps enhances reasoning accuracy. The automated pipeline ensures high throughput and reproducibility, setting a new standard for emotion understanding evaluation. Unlike traditional multiple-choice or subjective assessments, this approach provides fine-grained, objective metrics aligned with real-world social cognition. It bridges the gap between qualitative human judgments and scalable AI benchmarking, advancing the technical state-of-the-art in emotional intelligence measurement.
Novelty
This is the first systematic implementation of dialogue-driven emotion intensity ratings for large models, avoiding subjective bias inherent in crowd-sourced answers. The non-sum scoring mechanism and self-critique process are innovative, enabling more nuanced and accurate assessment of emotional comprehension. Compared to prior work like SECEU, this method improves discriminative power and robustness, establishing a new benchmark paradigm that emphasizes scene complexity and interpretive depth, pushing the frontier of AI emotional intelligence evaluation.
Limitations
- The question scenarios are manually curated, which may introduce bias and limit diversity. The scoring system may struggle with extreme or ambiguous emotional expressions, reducing sensitivity in such cases. The benchmark is primarily validated in English, and cross-cultural emotional nuances are not yet addressed. Model reasoning in complex scenarios remains imperfect, especially under limited context or computational constraints.
Future Work
Future efforts will expand to multi-modal scenarios incorporating images and audio, aiming for more holistic emotional understanding. Enhancing cultural and linguistic diversity in scene generation will improve generalizability. Integrating multi-task learning with emotional comprehension could further boost performance. Additionally, refining scoring algorithms to handle ambiguity and extreme cases will increase robustness. Long-term, the goal is to develop AI systems capable of nuanced social interactions across diverse real-world contexts.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在一家餐厅点菜，服务员需要理解你的心情和偏好，才能推荐合适的菜肴。情感智能就像是服务员的敏锐感知，知道你是开心、难过还是生气。EQ-Bench就像是一个测试，观察服务员在不同情境下能否准确判断你的心情。它用对话模拟你和服务员的交流，你告诉他一些细节，他要给每种情感打分，从0到10。比如，你说“我今天心情不好”，服务员要判断你有多难过或生气。这个测试帮助我们了解模型是否能像人一样理解别人的感受，从而让它们在实际生活中更善解人意。通过不断练习，模型会变得更擅长察觉别人的情绪，就像你学会了更好地读懂朋友的心思一样。
ELI14 Explained like you're 14
想象你和朋友在学校聊天，有时候你能感觉到他们的心情，比如开心、难过或生气。情感智能就像是你能理解他们的感受，知道他们为什么会那样想。EQ-Bench就像是一个游戏，里面有很多对话场景，你需要猜出朋友在那一刻心里在想什么，给每种感觉打分，从0到10。比如，朋友生气了，你要判断他有多生气，是不是很愤怒，还是只是有点不高兴。这个游戏帮助你学习如何更好地理解别人，知道他们的心情变化。通过不断练习，你会变得更善于察言观色，成为一个懂得关心别人的人。
Abstract
We introduce EQ-Bench, a novel benchmark designed to evaluate aspects of emotional intelligence in Large Language Models (LLMs). We assess the ability of LLMs to understand complex emotions and social interactions by asking them to predict the intensity of emotional states of characters in a dialogue. The benchmark is able to discriminate effectively between a wide range of models. We find that EQ-Bench correlates strongly with comprehensive multi-domain benchmarks like MMLU (Hendrycks et al., 2020) (r=0.97), indicating that we may be capturing similar aspects of broad intelligence. Our benchmark produces highly repeatable results using a set of 60 English-language questions. We also provide open-source code for an automated benchmarking pipeline at https://github.com/EQ-bench/EQ-Bench and a leaderboard at https://eqbench.com