When Robots Mishear Us: Mapping the Safety Risks of Voice-Controlled Embodied AI
This paper introduces a five-category ASR error simulation framework, evaluating its impact on embodied AI safety; findings show error type significantly influences risk levels.
Key Findings
Methodology
The study employs a five-category ASR error simulation framework—acoustic substitution, grammar confusion, punctuation errors, short-word omission, and noise addition—integrated with POEX and SafeAgentBench benchmarks. Using prompt-based generation via GPT-4 and CHIME-6 noise data, diverse corrupted inputs are created at varying intensities (WER levels). These inputs are systematically evaluated to assess their effects on safety metrics such as Acceptance Rate (AR), Executable Success Rate (ESR), and Safety Flips. The process involves collecting clean instructions, generating error-specific corrupted versions, and analyzing the influence on model behavior, including safety rejection and dangerous execution. The study also tests the efficacy of GPT-4-based correction strategies.
Key Results
- Results from POEX indicate that acoustic substitution and high-level noise significantly increase the acceptance of harmful instructions (AR from 55.33% to 67.33%) and the execution of unsafe plans (ESR from 35.33% to 36%), with safety flips rising to 27 cases. Post-correction, the risk associated with acoustic substitution decreases, but high noise remains problematic.
- In SafeAgentBench, ASR noise reduces task success rates (from 83% to 70%), demonstrating that noise impairs understanding and planning capabilities, especially under severe conditions.
- Different error types affect safety differently: acoustic substitution and noise addition directly raise dangerous behavior probabilities, while grammar confusion and punctuation mainly impair sentence coherence without significantly increasing risk. This highlights the importance of error type and semantic disruption in safety analysis.
Significance
This research emphasizes that ASR errors are a critical safety concern in embodied AI, extending beyond mere recognition accuracy. It provides a systematic framework for simulating and analyzing errors, guiding the development of more robust voice-controlled systems. The findings have implications for deploying AI in safety-critical environments, such as autonomous robots and assistive devices, where misinterpretations can lead to accidents or harm. By quantifying the impact of specific error types, the work informs future design strategies for safer AI interactions, addressing a long-standing challenge in human-robot communication.
Technical Contribution
The paper introduces a comprehensive five-category ASR error simulation framework, combining prompt-based generation with real-world noise data, enabling controlled and realistic error modeling. It integrates these errors into safety evaluation pipelines (POEX and SafeAgentBench), providing quantitative insights into how different errors influence safety metrics. The study also explores the effectiveness of GPT-4-based post-processing corrections, revealing their limitations. These contributions advance the state-of-the-art in robustness testing for embodied AI, bridging the gap between recognition accuracy and safety assurance, and establishing a new methodology for systematic risk analysis.
Novelty
This work is the first to systematically simulate and evaluate multiple controllable ASR error types within safety-critical benchmarks for embodied AI. Unlike prior research focusing solely on recognition accuracy or adversarial text attacks, it emphasizes the safety implications of real-world speech recognition errors. The integration of error simulation, safety evaluation, and correction strategies provides a novel comprehensive approach, offering new insights into the mechanisms by which input errors compromise safety and how to mitigate them.
Limitations
- The simulated errors, while diverse, may not fully capture the complexity of real-world acoustic environments, such as multi-speaker scenarios or dynamic noise conditions.
- The correction strategy relies heavily on GPT-4, which may not generalize well to unseen or more severe errors, limiting practical deployment.
- Experiments are conducted on benchmark datasets, which may not reflect the full variability of real-world applications, necessitating further validation in operational settings.
Future Work
Future research should incorporate real-world speech datasets with diverse acoustic conditions, develop adaptive error detection and correction algorithms, and explore multi-modal approaches combining visual and tactile cues. Extending evaluations to more complex tasks and environments will enhance robustness. Additionally, integrating safety-aware training methods and formal verification techniques can further mitigate risks posed by ASR errors in embodied AI systems.
AI Executive Summary
As embodied AI systems become increasingly integrated into public and industrial environments, voice control offers a natural and efficient interaction modality. However, automatic speech recognition (ASR) errors pose significant safety risks, especially in high-stakes scenarios. Traditional focus has been on improving recognition accuracy, but recent concerns highlight that even minor errors can lead to dangerous behaviors. This paper addresses this gap by systematically simulating five categories of ASR errors—acoustic substitution, grammar confusion, punctuation errors, short-word omission, and environmental noise—and evaluating their impact on safety using two benchmarks: POEX and SafeAgentBench.
The authors employ a combination of prompt-based generation with GPT-4 and real-world noise data from CHIME-6 to create diverse corrupted inputs at different severity levels. These inputs are then analyzed to measure how errors influence the likelihood of unsafe acceptance and execution of harmful instructions. Results reveal that acoustic substitution and high-intensity noise significantly increase safety flips and unsafe plan execution, with some correction strategies only partially mitigating these risks.
The findings demonstrate that ASR errors are not merely recognition inaccuracies but active safety risk factors. Different error types affect safety mechanisms differently: while grammar confusion mainly disrupts sentence coherence, acoustic errors directly increase dangerous behavior acceptance. These insights emphasize the need for robust error detection and correction in voice-controlled embodied AI, especially in safety-critical applications.
Overall, this work advances understanding of speech input risks, providing a comprehensive framework for simulating and evaluating ASR-induced safety hazards. It underscores that improving recognition accuracy alone is insufficient; safety must be integrated into the core design of voice-interactive AI systems. Future directions include real-world validation, multi-modal integration, and development of adaptive safety mechanisms, ensuring embodied AI can operate reliably and safely in complex environments.
Deep Dive
Abstract
We investigate whether automatic speech recognition (ASR) errors in user input can lead to unsafe outputs from Embodied AI (EAI) models. We find that ASR errors can lead to harmful instructions being accepted and executed by EAI models, thereby reducing safety. We simulate ASR errors and combine them with existing safety benchmarks (SafeAgentBench and POEX) to evaluate how different errors affect embodied AI safety. We find that some of them preserve semantic structure but increase harmful ambiguity, while others weaken the model refusal behaviour and allow unsafe plans to be generated and executed. We show that in some cases automatic correction of ASR errors can reduce the risk, but this is not always effective. Overall, we show that ASR errors lead to significant safety risks for embodied AI.