SafetyALFRED: Evaluating Safety-Conscious Planning of Multimodal Large Language Models

Multimodal large language models (MLLMs) have demonstrated remarkable reasoning and decision-making capabilities, leading to their widespread adoption as autonomous embodied agents in both simulated and physical interactive environments. They can translate high-level natural language instructions into executable plans. However, as MLLMs transition into these roles, a major concern is their ability to identify and proactively resolve safety hazards, i.e., observable environmental states that, if left uncorrected, pose risks of physical injury, property damage, or resource loss. Despite this need, prior safety benchmarks like ASIMOV, Multimodal Situational Safety, and MM-SafetyBench have largely focused on the recognition of hazards through question-answering (QA) tasks based on static images, videos, or scenarios. A critical gap remains in evaluating an agent’s ability to not only recognize safety hazards but also generate plans that mitigate them in a dynamic embodied setting.

Multimodal large language models are increasingly adopted as autonomous agents in interactive environments, yet their ability to proactively address safety hazards remains insufficient. Existing safety evaluations primarily focus on hazard recognition through disembodied question answering (QA) settings, neglecting the ability to actively mitigate risks in embodied contexts. To evaluate whether MLLMs can translate safety knowledge acquired from web-scale pre-training into concrete behavior, we formulate a new safety problem. Given a task instruction and a multimodal observation, the model must advance the assigned task while proactively generating a plan to rectify hazards that could cause immediate or future harm.

SafetyALFRED is the first study to extend the safety evaluation of multimodal LLMs from static QA to embodied contexts. Unlike existing benchmarks such as ASIMOV and MM-SafetyBench, SafetyALFRED not only focuses on hazard recognition but also emphasizes active risk mitigation, filling a critical gap in current research. We introduce an extension of the ALFRED benchmark for embodied instruction following, augmented with six carefully selected safety hazards that represent real-world risks in common kitchen settings. Using SafetyALFRED, we evaluate eleven MLLMs in two settings: (1) where the agent acts as a safety judge and identifies hazards in the scene; and (2) an embodied task where the agent completes the assigned task while immediately mitigating any safety hazards.

• �� SafetyALFRED builds on the ALFRED benchmark, extending it with six categories of real-world kitchen hazards.
• �� The study evaluates 11 state-of-the-art models from the Qwen, Gemma, and Gemini families on not only hazard recognition but also active risk mitigation through embodied planning.
• �� The experiments reveal a significant alignment gap: while models can accurately recognize hazards in QA settings, average mitigation success rates are low in comparison.
• �� We propose a multi-agent framework to decouple hazard recognition from mitigation, although this approach only slightly improves performance.

In our experiments, we used 30 kitchen environments and five task types in AI2Thor, involving object manipulation (move, stack, wash, heat, or cool), followed by placing the object at a final destination. We evaluate eleven state-of-the-art models from the Qwen, Gemma, and Gemini families on not only hazard recognition but also active risk mitigation through embodied planning. Our experimental results reveal a significant alignment gap: while models can accurately recognize hazards in QA settings, average mitigation success rates for these hazards are low in comparison.

Our experimental results show that while models perform well in QA tasks, they fail to effectively utilize their safety knowledge for actual behavior in embodied tasks. Even with metadata, open-weight models achieve less than 20% accuracy on average in all other categories, despite higher hazard identification rates in QA tasks. The multi-agent framework, while slightly improving performance, does not fully resolve the alignment issue between recognition and mitigation.

The application scenarios of SafetyALFRED include the development and deployment of autonomous robotic systems, particularly in situations where identifying and mitigating safety hazards in real-world environments is required. By introducing SafetyALFRED, the research demonstrates the capability gap in multimodal LLMs for recognizing and mitigating safety hazards in real-world kitchen environments. This finding is significant for academia and industry, emphasizing the need for more comprehensive safety evaluation methods.

The low mitigation success rate in embodied tasks indicates difficulties in planning and executing corrective actions, especially without metadata. Despite good performance in QA tasks, models fail to effectively utilize their safety knowledge for actual behavior in embodied tasks. The multi-agent framework, while slightly improving performance, does not fully resolve the alignment issue between recognition and mitigation. Future research directions include developing more effective model architectures to improve risk mitigation capabilities of multimodal LLMs in embodied tasks.