SocialReasonBench: A Video-QA Benchmark for Social Reasoning with Counterfactual Narrative Videos
SocialReasonBench evaluates social reasoning via branching narrative videos, revealing LMMs' shortcomings in counterfactual reasoning.
Key Findings
Methodology
SocialReasonBench uses Detroit: Become Human gameplay videos, employing a multi-agent data processing pipeline to generate social reasoning questions. This includes clip selection, signal alignment, and question generation, ensuring answers are based on game-state signals.
Key Results
- Experiments show current LMMs perform well on basic social understanding but poorly on counterfactual and causal reasoning, with accuracy only at 50%.
- Modality ablation reveals audio cues are crucial for affective and causal reasoning, while visual shortcuts are common errors.
- Diagnostic error analysis reveals models rely on incomplete modality cues, leading to reasoning traps.
Significance
This study fills the gap in existing social video reasoning benchmarks by introducing branching narrative videos, providing a new tool for evaluating models' reasoning capabilities in complex social contexts. It lays the groundwork for future socially assistive AI systems.
Technical Contribution
SocialReasonBench offers a new method for social reasoning evaluation through branching narrative structures, surpassing traditional benchmarks' linear narrative limitations, allowing alternative outcomes to be observed. It also develops a scalable multi-agent data processing pipeline.
Novelty
This is the first benchmark to use interactive narrative game videos for social reasoning evaluation, differing from traditional methods with fixed video clips by providing counterfactual results from branching narratives.
Limitations
- Current models perform poorly in counterfactual reasoning, often relying on visual shortcuts rather than deep social state reasoning.
- The benchmark relies on a specific game environment, which may limit its applicability in other contexts.
- Experiments did not cover all possible social scenarios.
Future Work
Future research could extend to other interactive narrative games, exploring reasoning capabilities in different social contexts. Additionally, new models could be developed to improve counterfactual reasoning performance.
AI Executive Summary
Recent advances in multimodal models have significantly improved video understanding. However, existing benchmarks typically rely on videos with a single observed trajectory, making it difficult to determine whether models truly understand social dynamics. SocialReasonBench evaluates social reasoning capabilities through branching narrative videos, using gameplay videos from Detroit: Become Human to construct a multiple-choice QA benchmark. This benchmark covers seven reasoning dimensions, including intent recognition, emotional empathy, moral dilemma, counterfactual reasoning, and causal antecedent. Experimental results show that current models perform well on basic social understanding but struggle with counterfactual and causal reasoning. Further ablation and diagnostic error analyses reveal that models often depend on incomplete modality cues, leading to reasoning traps. This study lays the groundwork for future socially assistive AI systems and provides a new tool for evaluating models' reasoning capabilities in complex social contexts.
Deep Analysis
Background
Recent advances in multimodal models have significantly improved video understanding. However, existing benchmarks typically rely on videos with a single observed trajectory, making it difficult to determine whether models truly understand social dynamics. Social reasoning is crucial for downstream applications such as healthcare and socially assistive AI systems.
Core Problem
Existing social video reasoning benchmarks rely on linear narrative videos, making it difficult to evaluate models' counterfactual reasoning capabilities. Models need to infer latent social dynamics, including beliefs, intentions, emotions, and interpersonal motives.
Innovation
SocialReasonBench evaluates social reasoning capabilities through branching narrative videos, using gameplay videos from Detroit: Become Human to construct a multiple-choice QA benchmark. It covers seven reasoning dimensions, providing counterfactual results from branching narratives.
Methodology
- �� Use a multi-agent data processing pipeline to select socially meaningful clips
- �� Align answer labels with game-state signals
- �� Generate theory-guided questions with diagnostic distractors
Experiments
Experiments evaluated 8 representative LMMs using a zero-shot setting. Results show models perform well on basic social understanding but poorly on counterfactual and causal reasoning. Modality ablation reveals audio cues are crucial for affective and causal reasoning.
Results
Experimental results show current models perform well on basic social understanding but poorly on counterfactual and causal reasoning. Diagnostic error analysis reveals models rely on incomplete modality cues, leading to reasoning traps.
Applications
The benchmark can be used to evaluate socially assistive AI systems' reasoning capabilities, helping develop more socially intelligent models. It can also be used for education and training, helping people understand complex social contexts.
Limitations & Outlook
Current models perform poorly in counterfactual reasoning, often relying on visual shortcuts rather than deep social state reasoning. The benchmark relies on a specific game environment, which may limit its applicability in other contexts.
Plain Language Accessible to non-experts
Imagine you're playing an interactive game where every choice you make affects the story's outcome. SocialReasonBench is like a game that evaluates models' social reasoning capabilities through videos. Models need to understand characters' intentions and emotions, predicting the outcomes of different choices, just like a player making decisions in a game. This benchmark helps us evaluate models' performance in complex social contexts, just like evaluating a player's performance in a game.
ELI14 Explained like you're 14
Imagine you're playing a super cool game where every choice changes the story's outcome. SocialReasonBench is like that game, evaluating AI's social reasoning skills through videos. AI needs to understand other characters' intentions and emotions, predicting the outcomes of different choices. This benchmark helps us evaluate AI's performance in complex social contexts, just like evaluating a player's performance in a game. Isn't that fun?
Glossary
Counterfactual Reasoning
Inferring what would happen under alternative choices.
Used to evaluate models' reasoning capabilities in branching narratives.
Social Dynamics
Changes in trust, emotions, and motives between characters.
Models need to infer latent social dynamics.
Multimodal Models
Models that process multiple data types, such as video and text.
Used for video understanding and social reasoning.
Branching Narrative
Stories that develop based on player choices.
Used to construct counterfactual reasoning benchmarks.
Diagnostic Distractors
Options designed to test models' reasoning errors.
Used to evaluate models' reasoning capabilities.
Open Questions Unanswered questions from this research
- 1 Models still need improvement in counterfactual reasoning, especially in complex social contexts.
- 2 Exploration of applying this benchmark to other interactive games is needed.
Applications
Immediate Applications
Education and Training
Helps students understand complex social contexts, improving social intelligence.
Long-term Vision
Socially Assistive AI
Develop more socially intelligent AI systems for healthcare and interpersonal interactions.
Abstract
Recent advances in Large Multimodal Models (LMMs) have greatly improved video understanding, yet their ability to reason about human-centered social situations remains limited. Existing benchmarks typically rely on videos with a single observed trajectory, making it difficult to determine whether models truly understand social dynamics or merely exploit recurring narrative patterns. We introduce SocialReasonBench, a video multiple-choice QA benchmark for evaluating socially grounded reasoning in scenarios derived from interactive narratives. Built from gameplay videos of Detroit: Become Human, the benchmark leverages branching storylines where player decisions lead to alternative social outcomes that can be checked against the game's own script, flowchart, and recorded branches. We develop a multi-agent curation pipeline that localizes socially meaningful clips, grounds answer labels in game-state signals, and generates theory-guided questions with diagnostic distractors. SocialReasonBench covers seven reasoning dimensions, including intent recognition, emotional empathy, moral dilemma, counterfactual reasoning, and causal antecedent. Experiments on contemporary LMMs show that models perform reasonably well on basic social understanding but struggle with counterfactual and causal reasoning. Further ablation and diagnostic error analyses reveal that models often depend on incomplete modality cues and fall into reasoning traps such as visual shortcuts, highlighting a gap between observable event recognition and deeper reasoning over latent social states.