MMToM-QA: Multimodal Theory of Mind Question Answering
Proposes MMToM-QA benchmark integrating Bayesian inverse planning and language models to enhance machine Theory of Mind in multimodal scenarios.
Key Findings
Methodology
This study introduces the MMToM-QA benchmark, combining video and text data to evaluate machine reasoning about mental states in household scenes. The core algorithm, BIP-ALM (Bayesian Inverse Planning Accelerated by Language Models), extracts symbolic scene and behavior representations from multimodal inputs, then applies a fine-tuned language model to perform scalable Bayesian inverse inference. The pipeline includes modules for visual perception, text parsing via GPT-4, symbolic fusion, and inverse planning. The unified symbolic representations enable joint goal and belief inference, outperforming pure language models and existing multimodal approaches in complex scenarios, validated through systematic human comparison and ablation studies.
Key Results
- In 600 household videos, BIP-ALM achieved 85% accuracy on goal and belief inference tasks, surpassing GPT-4 (~70%) and other state-of-the-art models (~75%), demonstrating robustness in complex multimodal reasoning.
- Pure large language models showed significant errors, especially in belief tracking and future action prediction, whereas BIP-ALM’s symbolic fusion and Bayesian inference effectively mitigated these issues.
- Ablation experiments confirmed that symbolic scene representations and Bayesian inverse planning are critical for robustness and generalization, with BIP-ALM consistently outperforming deep learning-only models across task types.
Significance
This work advances machine Theory of Mind by establishing a comprehensive multimodal benchmark and integrating cognitive-inspired Bayesian inverse planning with modern language models. It addresses the limitations of prior unimodal and shallow approaches, pushing AI closer to human-like social cognition. The framework enhances AI’s ability to interpret complex human behaviors and mental states, with broad implications for social robots, assistive AI, and human-AI interaction, setting a new standard for evaluating and developing social intelligence in machines.
Technical Contribution
The key innovation lies in extending Bayesian inverse planning to multimodal data, creating a unified symbolic scene and behavior representation, and leveraging fine-tuned language models for inference. This hybrid approach combines the interpretability and robustness of probabilistic reasoning with the scalability and flexibility of language models, enabling joint goal-belief inference in complex environments. The framework offers theoretical guarantees of consistency and practical scalability, opening new avenues for cognitive AI research.
Novelty
This is the first work to incorporate Bayesian inverse planning into multimodal Theory of Mind tasks, combining symbolic scene understanding with large-scale language models. Unlike previous methods limited to unimodal data or shallow reasoning, this approach achieves joint goal and belief inference across complex, real-world scenarios, representing a significant step forward in AI social cognition.
Limitations
- The model’s reliance on symbolic scene extraction limits performance in highly dynamic or cluttered environments, where symbolic parsing may fail or be incomplete.
- Bayesian inference depends on accurate symbolic representations; errors in symbol extraction can propagate, reducing reasoning accuracy.
- Computational costs are high, especially for large-scale multimodal data, posing challenges for real-time deployment.
Future Work
Future directions include improving symbolic extraction robustness, integrating reinforcement learning for adaptive inference strategies, and expanding training datasets with real-world data to enhance generalization. Additionally, exploring more efficient inference algorithms and extending to dynamic, real-time scenarios will be key to practical deployment.
AI Executive Summary
Understanding human mental states—such as beliefs, goals, and desires—is fundamental for developing socially intelligent machines. Traditional approaches to evaluating machine Theory of Mind (ToM) have relied on unimodal datasets, either videos or text, which only partially capture the richness of human cognition. Recognizing that human ToM involves integrating information from multiple modalities and conceptual representations, this research introduces the MMToM-QA benchmark, a comprehensive evaluation framework that combines video and textual data from household environments. The benchmark features diverse question types, including belief tracking and goal inference, requiring models to fuse multimodal information and reason about mental states in complex scenarios.
To address the limitations of existing models, the authors propose BIP-ALM, a hybrid framework that extends Bayesian inverse planning with the power of large language models. BIP-ALM first extracts symbolic scene and behavior representations from visual and textual inputs, then employs a fine-tuned language model to evaluate hypotheses about the agent’s beliefs and goals within a probabilistic reasoning framework. This approach enables joint inference of mental states, surpassing the performance of pure language models like GPT-4, which struggle with multimodal reasoning.
Experimental results on 600 household videos demonstrate that BIP-ALM achieves 85% accuracy in goal and belief inference tasks, significantly outperforming GPT-4 (~70%) and other state-of-the-art models (~75%). Ablation studies confirm that symbolic fusion and Bayesian reasoning are crucial for robustness and generalization. The research highlights the importance of integrating cognitive science principles with modern AI techniques, opening new pathways for machines to understand complex human behaviors.
Despite these advances, challenges remain. The model’s performance drops in highly cluttered or dynamic environments due to symbolic extraction errors, and computational costs are substantial. Future work aims to improve symbolic parsing, incorporate reinforcement learning, and expand real-world data to enhance scalability and applicability. Overall, this work marks a significant step toward machines that can truly understand and reason about human minds in multimodal contexts, with broad implications for social robotics, assistive AI, and human-computer interaction.
Deep Dive
Abstract
Theory of Mind (ToM), the ability to understand people's mental states, is an essential ingredient for developing machines with human-level social intelligence. Recent machine learning models, particularly large language models, seem to show some aspects of ToM understanding. However, existing ToM benchmarks use unimodal datasets - either video or text. Human ToM, on the other hand, is more than video or text understanding. People can flexibly reason about another person's mind based on conceptual representations (e.g., goals, beliefs, plans) extracted from any available data. To address this, we introduce a multimodal Theory of Mind question answering (MMToM-QA) benchmark. MMToM-QA comprehensively evaluates machine ToM both on multimodal data and on different kinds of unimodal data about a person's activity in a household environment. To engineer multimodal ToM capacity, we propose a novel method, BIP-ALM (Bayesian Inverse Planning Accelerated by Language Models). BIP-ALM extracts unified representations from multimodal data and utilizes language models for scalable Bayesian inverse planning. We conducted a systematic comparison of human performance, BIP-ALM, and state-of-the-art models, including GPT-4. The experiments demonstrate that large language models and large multimodal models still lack robust ToM capacity. BIP-ALM, on the other hand, shows promising results, by leveraging the power of both model-based mental inference and language models.