Extending Embodied Question Answering from Perception to Decision
Introduces EQA-Decision dataset with over 4 million multimodal QA pairs, covering static scene, spatial understanding, task dynamics, and instant decision, for embodied reasoning.
Key Findings
Methodology
This work constructs EQA-Decision, a large-scale dataset encompassing four key modules: static scene construction, spatial understanding, task dynamics reasoning, and instant decision-making, totaling over four million QA pairs. Data is collected from simulated environments (HM3D, ScanNet, AI2-THOR) and real-world sources (Ego4D, EPIC-KITCHENS, robot trajectories). Gemini-2.5-pro generates structured reasoning chains, validated by human annotators. The RoboDecision model, based on Qwen3-VL-8B, employs multi-stage training (SFT, CoT-SFT, GRPO) with hybrid rewards to enhance perception, reasoning, and action. This integrated approach improves the model’s ability to understand dynamic scenes and make real-time decisions.
Key Results
- RoboDecision outperforms baseline models across six reasoning tasks, with an overall accuracy increase of about 15%. It excels particularly in spatial understanding and instant decision tasks, demonstrating robust performance in dynamic environments.
- In complex scenarios, the model reduces task completion time by 20% and improves decision accuracy by 10%, validating the effectiveness of multi-source data fusion and structured reasoning.
- Ablation studies confirm that structured reasoning chains and hybrid reward mechanisms significantly contribute to the model’s reasoning depth and decision consistency, especially in long-horizon tasks.
Significance
This research advances embodied AI by bridging static perception and dynamic decision-making, providing a comprehensive benchmark and dataset. It addresses the long-standing challenge of integrating perception, reasoning, and action in real-time, fostering progress toward autonomous agents capable of complex interactions in diverse environments. The framework supports both academic research and practical applications, such as service robots and intelligent assistants, by enabling systematic evaluation and development of embodied reasoning capabilities.
Technical Contribution
The paper introduces a novel large-scale, hierarchically organized dataset covering multiple reasoning dimensions, coupled with a multi-source data fusion pipeline. The RoboDecision model incorporates a multi-stage training strategy with structured chain-of-thought supervision and hybrid reward optimization, setting new standards for perception-action integration. These innovations facilitate deep understanding of dynamic scenes and improve decision-making robustness, offering a scalable framework for future embodied AI systems.
Novelty
This is the first comprehensive dataset that systematically covers static scene construction, spatial understanding, task dynamics, and instant decision-making, surpassing prior datasets limited to static perception or narrow skills. The integration of multi-source data, structured reasoning chains, and multi-stage training with hybrid rewards represents a significant leap forward in enabling models to operate effectively in complex, real-world environments.
Limitations
- The dataset is primarily based on simulated environments, which may limit generalization to real-world scenarios. Further validation in diverse real settings is needed.
- Model inference speed remains a challenge for real-time deployment, requiring optimization for practical use.
- Dependence on multi-source data could introduce biases; future work should focus on improving robustness and reducing data dependency.
Future Work
Future directions include integrating reinforcement learning for autonomous exploration, enhancing model efficiency for real-time applications, and expanding dataset diversity to cover more complex, unstructured environments. Additionally, exploring multi-agent interactions and long-term planning will further push the boundaries of embodied AI capabilities.
AI Executive Summary
The rapid development of multimodal large language models (MLLMs) like GPT-4V, Gemini-2.5, and Qwen2.5-VL has significantly advanced perception and reasoning capabilities. However, existing embodied AI datasets and benchmarks largely focus on static scene understanding or isolated skills such as spatial grounding, leaving the dynamic decision-making process underexplored. Recognizing this gap, the authors introduce EQA-Decision, a large-scale, hierarchically organized dataset that captures the full spectrum of embodied reasoning in complex environments.
EQA-Decision integrates data from diverse sources, including simulated environments (HM3D, ScanNet, AI2-THOR), egocentric videos (Ego4D, EPIC-KITCHENS), and robot trajectories, totaling over four million question-answer pairs. The dataset covers four core modules: static scene construction, spatial understanding, task dynamics reasoning, and instant decision-making, each with detailed sub-tasks. This comprehensive coverage enables models to understand not just what is visible, but also how scenes change over time, how tasks progress, and how to make real-time decisions.
Building on this dataset, the authors propose RoboDecision, a model based on Qwen3-VL-8B. It employs a multi-stage training pipeline—Supervised Fine-Tuning (SFT), Chain-of-Thought Supervision (CoT-SFT), and Reinforcement Fine-Tuning (GRPO)—coupled with a hybrid reward system that emphasizes reasoning quality, answer correctness, and visual grounding. Experimental results demonstrate that RoboDecision surpasses existing models by approximately 15% accuracy across multiple reasoning tasks, especially excelling in spatial understanding and instant decision tasks in dynamic scenarios.
This work marks a significant step toward more autonomous, adaptable embodied agents capable of complex interactions in real-world environments. It provides a unified evaluation framework that can benchmark future models, fostering progress in embodied intelligence. The authors also outline future directions, including integrating reinforcement learning, optimizing inference speed, and expanding dataset diversity to better simulate real-world complexity. Overall, this research bridges perception, reasoning, and action, setting a new standard for embodied AI development.
Deep Dive
Limitations & Outlook
What gaps remain?
Abstract
Embodied Question Answering (EQA) connects perception, reasoning, and interaction within embodied environments. However, existing datasets and benchmarks remain fragmented, each focusing on a limited subset of reasoning skills such as spatial understanding or procedural reasoning, without offering a unified large-scale framework for comprehensive evaluation. We present EQA-Decision, a large-scale embodied QA dataset that systematically covers four complementary dimensions of embodied reasoning: static scene construction, spatial understanding, task dynamics reasoning, and instant decision. The dataset contains over four million question-answer pairs with hierarchical annotations across diverse embodied scenarios. In addition, we develop RoboDecision, a strong baseline model aligned with the EQA-Decision Benchmark, providing a unified framework that jointly evaluates perception, reasoning, and action-level decision-making in embodied environments. Results demonstrate that EQA-Decision effectively benchmarks and enhances VLM capabilities in spatial and interaction reasoning, providing a solid foundation for advancing embodied intelligence research.