Semantic Flip: Synthetic OOD Generation for Robust Refusal in Embodied Question Answering and Spatial Localization

The development of vision-language models (VLMs) such as CLIP, BLIP, and Flamingo has significantly advanced embodied AI tasks, including question answering and spatial reasoning. These models leverage large-scale pretraining on image-text pairs, enabling flexible understanding and reasoning. Notable works like ReMEmbR and META-MEMORY introduced long-term memory modules to improve navigation and question answering in complex environments. However, despite these advances, models tend to produce overconfident answers even when queries are unsupported or ambiguous, risking misinformation. Traditional detection methods rely on supervised datasets with predefined categories, which are impractical in open-world scenarios. Recent efforts in out-of-distribution detection focus on post hoc scoring or ensemble methods, but these are often computationally expensive and less effective in multi-modal settings. This context motivates the need for scalable, annotation-free approaches that can teach models to reject unsupported inputs reliably, especially in safety-critical applications like robotics and autonomous navigation.

The core challenge in embodied AI is enabling models to accurately identify when a user query is unanswerable or unsupported by visual evidence. Existing systems often respond confidently to unsupported questions, risking misinformation or unsafe actions. Supervised methods require extensive labeled datasets, which are costly and limited in scope. Prompt engineering offers some flexibility but remains sensitive to wording and lacks robustness. The fundamental problem is how to train models to recognize ungrounded or unsupported inputs without relying on large, curated out-of-distribution datasets. Achieving this requires a method that can generate diverse, realistic unanswerable examples during training, enabling models to generalize to unseen unsupported queries in real-world scenarios.

Semantic Flip introduces a novel data augmentation strategy for embodied refusal. It independently corrupts either the query (Q-FLIP) or visual memory (V-FLIP) to generate auxiliary out-of-distribution samples without external annotations. Q-FLIP uses large language models to generate unanswerable questions by paraphrasing or introducing false premises, while V-FLIP employs object detection and inpainting to erase visual grounding. These synthetic pairs are used to train a lightweight rejection head on top of a frozen VLM, enabling the system to distinguish answerable from unanswerable inputs effectively. The approach is simple, scalable, and compatible with existing models, avoiding costly fine-tuning. Additionally, the paper introduces SPACEREJECT, a new benchmark for long-term spatial refusal, extending existing spatial QA datasets to include unanswerable queries, thus evaluating the model's robustness in real-world scenarios.

• �� Start with a dataset of answerable (Q, V, A) triplets from embodied tasks.
• �� Generate Q-FLIP samples by prompting large language models (e.g., GPT-3) to produce questions that cannot be answered from the given visual context, simulating referential errors, false premises, or subjective queries.
• �� Generate V-FLIP samples by detecting target objects using models like Grounding-DINO, then removing these objects via inpainting methods such as LaMa, creating visual memory pairs lacking the necessary grounding.
• �� Extract joint embeddings of original and synthetic pairs using a frozen VLM encoder (e.g., Qwen2.5-VL).
• �� Train a small 3-layer MLP classifier on these embeddings to output a rejection score, with labels 0 for answerable, 1 for unanswerable.
• �� During inference, input (Q, V) pairs are encoded, and the rejection head predicts whether to answer or abstain based on a threshold.
• �� The entire pipeline integrates seamlessly into existing embodied systems without modifying the core VLM, enabling scalable deployment across tasks.

• �� Conduct evaluations on ABSTAINEQA, a benchmark with answerability annotations, and SPACEREJECT, designed for spatial localization with unanswerable queries.
• �� Use a frozen 7B or 32B Qwen model as the backbone, training only the lightweight rejection module.
• �� Compare against prompt-based baselines, including Coarse and Fine prompts, and prompt-enhanced Chain-of-Thought prompting.
• �� Metrics include F1, balanced accuracy, recall, and specificity, with ablation studies varying the scale of synthetic samples.
• �� Results show that Semantic Flip consistently outperforms prompt-based methods, with significant improvements in OOD detection and refusal accuracy, especially in long-term spatial tasks.
• �� Additional analyses include per-category OOD recall, robustness to different corruptions, and generalization across datasets.

• �� The method achieves an F1 of 0.7110 on ABSTAINEQA with a 7B model, surpassing larger prompt-based models.
• �� On SPACEREJECT, the combined Q-FLIP and V-FLIP approach reaches an F1 of 0.9559, demonstrating high robustness in spatial refusal.
• �� Ablation results indicate both components contribute significantly, with Q-FLIP providing most of the signal.
• �� The approach maintains high performance without retraining the entire VLM, reducing computational costs and simplifying deployment.
• �� The synthetic data effectively teaches models to reject unsupported inputs, generalizing well across different tasks and environments.

• �� Immediate: deployment in embodied robots for navigation and question answering, ensuring safety by rejecting unsupported queries.
• �� Long-term: development of autonomous agents with reliable self-assessment capabilities, reducing reliance on large labeled datasets, and enabling scalable, trustworthy AI systems in diverse real-world scenarios.

• �� Synthetic samples are generated via single-modality corruptions, which may not cover all types of invalid inputs, limiting the rejection scope.
• �� Visual inpainting can introduce artifacts, potentially affecting feature quality and robustness.
• �� Effectiveness in highly dynamic, cluttered, or ambiguous environments remains to be validated, requiring further robustness improvements.