Neurosymbolic Embodied Agents
Neurosymbolic embodied agent combines visual exploration and symbolic planning, achieving over 90% success in household tasks.
Key Findings
Methodology
This paper introduces a two-phase neurosymbolic architecture for embodied agents tackling long-horizon household tasks. Phase I employs a vision-language model (VLM) coupled with an exploration harness to actively acquire goal-relevant predicates and object bindings from egocentric observations, forming a symbolic initial state. Phase II involves symbolic planning constrained by a PDDL transition model, where decoding is restricted to tokens representing applicable actions. Monte Carlo tree search (MCTS) evaluates executable continuations using a domain-independent heuristic, integrating procedural priors from language models with explicit state transitions. This approach guarantees that generated plans are executable under the symbolic model, with transfer to the environment conditioned on correct visual grounding. Experiments on VirtualHome and ALFWorld with models from 4B to 27B parameters show success rates exceeding 90%, outperforming direct visual policies and unstructured reasoning methods, demonstrating the effectiveness of combining structured symbolic planning with visual exploration.
Key Results
- In VirtualHome, success rates reach 99.0% with Qwen3.5-27B, and in ALFWorld, 95.5%, both significantly higher than baseline models (~60%), illustrating the advantage of symbolic constraints in long-horizon planning.
- Models of different sizes (4B, 9B, 27B) show success rate improvements, with even the smallest (4B) achieving 94.5%, outperforming direct visual policies by about 32 percentage points, confirming the efficiency of the structured approach.
- Combining symbolic constraints with search elevates success in ALFWorld from below 33% (single methods) to over 95%, highlighting the strong synergy between constraints and search in complex tasks.
Significance
This work addresses a fundamental challenge in embodied AI: ensuring task plans derived from perceptual inputs are feasible and executable over long horizons. By integrating symbolic reasoning with visual exploration, the proposed framework effectively reduces the gap between perception and action, enabling robots to perform complex household tasks reliably. The approach offers a scalable, interpretable, and verifiable planning mechanism that can be extended to real-world applications, such as assistive robots and autonomous service agents. It also advances the theoretical understanding of combining deep perceptual models with formal symbolic planning, paving the way for more robust and explainable autonomous systems.
Technical Contribution
The core technical innovation lies in embedding PDDL-based applicability constraints directly into the language model decoding process, ensuring each generated action is valid under the symbolic transition model. The use of state-dependent constrained decoding combined with MCTS for long-term evaluation creates a robust planning pipeline that guarantees plan executability by construction. This method bridges the gap between neural perceptual models and symbolic reasoning, enabling end-to-end, verifiable long-horizon planning without requiring additional training or external validation. The architecture’s modularity allows flexible integration of different models and heuristics, setting a new standard for neurosymbolic planning in embodied AI.
Novelty
This is the first comprehensive integration of state-dependent constrained decoding with MCTS guided by a formal PDDL transition model within a vision-language framework for long-horizon tasks. Unlike prior works that treat symbolic planning as a downstream or separate module, this approach embeds applicability constraints directly into the generation process, ensuring every action is valid at the symbolic level. The explicit separation of perception and planning phases, combined with a formal guarantee of plan validity, represents a significant departure from end-to-end models that lack such guarantees. This novel architecture effectively leverages the strengths of deep learning and symbolic reasoning, setting a new paradigm for reliable autonomous planning.
Limitations
- The method heavily relies on accurate visual grounding during Phase I; errors in perception can lead to incorrect symbolic states, which in turn affect plan validity, especially in cluttered or dynamic environments.
- The approach assumes deterministic environment dynamics and perfect symbolic bindings, which may not hold in real-world scenarios with noise, partial observability, or unforeseen changes.
- Computational costs increase with model size and search budget, potentially limiting real-time applications, and scalability to highly complex tasks remains to be validated.
Future Work
Future research will focus on enhancing perception robustness, integrating online learning for dynamic environments, and extending the framework to real-world robotic platforms. Incorporating probabilistic models to handle uncertainty, developing adaptive heuristics for large-scale planning, and exploring multi-agent collaboration are promising directions. Additionally, efforts to automate symbolic knowledge acquisition and improve scalability will be crucial for deploying such systems in practical settings.
AI Executive Summary
Long-term autonomous task execution in household environments remains a significant challenge in embodied AI. Existing vision-language models (VLMs) can generate plausible action sequences from natural language goals, but they lack guarantees of executability, often producing plans that violate environment dynamics or act on incorrectly grounded entities. This disconnect between perception and control hampers reliable deployment of autonomous agents in real-world scenarios.
To address this, the authors propose a neurosymbolic framework that decomposes the problem into two interconnected phases: exploration and planning. In the first phase, a vision-language model actively explores the environment, guided by a learned exploration harness, to acquire goal-relevant predicates and object bindings from egocentric observations. This process results in a symbolic initial state that accurately reflects the environment’s key facts, reducing uncertainty and grounding the subsequent planning process.
The second phase involves symbolic planning constrained by a formal PDDL (Planning Domain Definition Language) transition model. Here, the language model’s decoding process is restricted at each step to generate only actions that satisfy the preconditions, ensuring validity. To evaluate long-horizon consequences, the system employs Monte Carlo tree search (MCTS) combined with domain-independent heuristics, balancing procedural priors from the language model with explicit state transition evaluation. This guarantees that every generated plan is executable by construction, provided the initial symbolic state is accurate and the environment’s transition model is sound.
Extensive experiments on VirtualHome and ALFWorld benchmarks demonstrate the approach’s effectiveness. Success rates exceed 90% across various model sizes, with the 27B parameter model reaching 99% in VirtualHome. Notably, the method outperforms direct visual policies by over 30 percentage points, even with smaller models, highlighting the efficiency of structured symbolic reasoning. The combination of constraints and search proves highly synergistic, with the integrated approach solving over 95% of tasks in ALFWorld, compared to less than a third when using either component alone.
Beyond performance, the framework reduces the number of generated tokens and visual inputs needed, lowering inference costs significantly. It also localizes residual failures primarily to perception errors rather than plan generation, indicating robustness in symbolic reasoning. This work advances the state-of-the-art in neurosymbolic embodied AI, providing a scalable, verifiable, and interpretable approach for complex, long-horizon tasks. Future directions include extending to real-world robots, handling uncertainty, and integrating online learning to adapt to dynamic environments, paving the way for autonomous agents capable of reliable, long-term operation in human-centric settings.
Deep Analysis
Background
随着深度学习和多模态感知技术的快速发展，机器人自主任务规划逐步从简单的反应式行为演变为复杂的长远计划。早期工作如SLAM（同步定位与地图构建）实现了环境的空间建模，但在多目标、多步骤任务中仍存在环境动态变化和信息不完整的难题。近年来，符号推理（如PDDL）被引入机器人系统，用于确保计划的逻辑合理性，但其在高维感知环境中的应用受到感知误差和环境不确定性的限制。视觉语言模型（如GPT-4、Qwen系列）具备理解视觉和自然语言的能力，潜在地支持端到端的自主规划，但缺乏严格的环境动态保证。现有方法多依赖单一模型，难以兼顾感知的灵活性和推理的可靠性，导致长远任务中计划失效频繁。为此，融合符号推理与深度感知的神经符号方法逐渐成为研究热点，旨在弥补纯深度模型在可执行性上的不足，推动自主系统向更高层次发展。
Core Problem
长远家庭任务的核心难题在于感知、推理与规划的紧密耦合。感知阶段容易受到遮挡、误识别等影响，导致环境状态的误判；而推理阶段依赖准确的符号状态，任何感知误差都可能引发计划偏差。传统端到端模型在面对复杂场景时，常常生成无法实际执行的动作序列，严重限制了其应用潜力。解决这一问题的关键在于如何在保证感知灵活性的同时，确保生成的行动计划具有严格的可行性和验证性。尤其是在长远任务中，动作的顺序和环境状态的正确性至关重要，任何偏差都可能导致任务失败。为此，需要一种机制，既能高效感知环境，又能在符号层面保证计划的合理性和可行性，从而实现自主机器人在复杂环境中的可靠执行。
Innovation
本研究的核心创新在于提出一种两阶段的神经符号架构，有效解决感知与推理的耦合难题：
1) 探索阶段：利用预训练的视觉语言模型（如Qwen3.5）结合探索控制器，主动从不同视角检测目标对象和关系，形成符号化的环境描述，避免全面重建场景，从而减少感知误差。
2) 规划阶段：基于符号状态和PDDL定义的动作模型，采用受约束的解码策略，确保每个生成的动作都满足预条件。利用蒙特卡洛树搜索（MCTS）结合启发式评估长远后果，搜索最优行动序列。
3) 形式保证：每个计划在符号模型下都具备可行性，且在环境中转移时依赖正确的视觉地面化，实现符号与感知的无缝结合。这一创新突破了传统端到端模型在复杂环境中易产生不可执行计划的难题，为符号推理在高维感知环境中的应用提供了新范式。
Methodology
- �� 第一阶段（探索）：
- 输入：目标描述、机器人当前观察图像。
- 过程：
- 利用预训练的视觉语言模型（如Qwen3.5）与探索控制器合作，从不同视角主动检测目标相关的对象和关系。
- 通过多视角观察和 grounded interactions，提取目标谓词和实例绑定，形成符号化的环境状态。
- 输出：符号化的初始状态（谓词和实例绑定），为后续规划提供基础。
- �� 第二阶段（规划）：
- 输入：符号化的环境状态、目标描述。
- 过程：
- 利用PDDL定义动作的预条件和效果，结合符号状态限制动作空间。
- 在解码过程中，动态限制生成的动作符号，使其符合预定义的动作预条件。
- 采用受约束的蒙特卡洛树搜索（MCTS）结合启发式评估，搜索长远的行动序列。
- 输出：符合环境动态、可执行的行动计划（动作序列）。
- �� 核心机制：
- 受约束解码：在每个动作生成点限制生成符号，确保动作合法。
- 计划评估：利用启发式评估长远目标达成可能性，结合搜索优化。
- 转移保证：符号模型的正确性确保计划在符号层面可行，环境转移在视觉地面化后验证。
Experiments
- �� 数据集：使用VirtualHome和ALFWorld两个虚拟家庭环境，涵盖厨房、客厅、卧室等多场景，任务包括物品放置、物体检查、清洁等。
- �� 模型：采用Qwen3.5系列（4B、9B、27B参数）作为基础模型，比较不同规模的性能表现。
- �� 基线：直接视觉策略（单步预测动作）、纯推理模型（无符号约束）、不同搜索策略（贪婪、无约束MCTS、受约束MCTS）进行对比。
- �� 评估指标：主要为任务成功率（达到目标状态），同时统计生成的Token数、视觉输入次数和失败归因。
- �� 超参数：探索步骤限制（VirtualHome为20步，ALFWorld为55步），解码温度设为零或模型推荐值，搜索预算根据任务复杂度调整。
- �� 其他：进行消融实验验证符号约束、搜索策略和模型规模的影响。
Results
- �� 在两个环境中，基于Qwen3.5-27B的神经符号代理成功率分别达到99.0%（VirtualHome）和95.5%（ALFWorld），远超不结合符号规划的模型（成功率约在60%），显示符号约束极大提升了长远任务的可行性和效率。
- �� 不同模型规模的实验表明，即使是最小的4B模型，也能达到94.5%的成功率，较直接视觉策略提升约32个百分点，验证了结构化符号推理在不同规模模型中的适应性。
- �� 结合符号约束和搜索策略，任务在ALFWorld中的成功率由不足33%（单一方法）提升至超过95%，显示两者的互补性极强，极大改善了长远任务的执行能力。
Applications
- �� 立即应用：该方法可用于智能家居机器人，实现自主的物品整理、环境维护等任务，前提是具备基本的视觉感知和符号识别能力，提升家庭自动化水平。
- �� 长远愿景：未来可扩展到服务机器人、救援机器人等复杂场景，通过增强感知和符号推理能力，实现更高层次的自主决策和长远规划，推动智能家居和公共服务的普及。
Plain Language Accessible to non-experts
想象你在厨房准备做饭。你需要先看看冰箱里有什么，锅在哪儿，火是不是开着，然后再决定下一步。以前的机器人就像是一个只会跟着指令走的机械，没有自己的观察能力，只知道你告诉它做什么，但如果它没看到冰箱里的东西，就可能做错事。而这篇论文的方法，就像是给机器人装上了“眼睛”和“脑袋”。它会主动观察厨房，从不同角度看东西，记住重要的细节，然后用这些信息制定一份详细的做饭计划，确保每一步都能完成。这样一来，机器人就能像人一样，既能观察环境，又能合理规划，避免做出不可能完成的任务。
ELI14 Explained like you're 14
想象你在厨房里准备做饭。你会先看看冰箱里有什么，确认锅在哪儿，然后决定下一步要做什么。以前的机器人就像是个只会按顺序做事的机械，没有自己的观察能力，它只知道你告诉它做什么，但不知道厨房里的实际情况。这篇论文的方法就像是给机器人装上了“眼睛”和“脑袋”。它会主动观察厨房，记住哪些东西在哪里，哪些可以用，然后用这些信息写出一份详细的菜谱，确保每一步都能顺利完成。这样一来，机器人就能更聪明、更可靠地完成复杂的任务，不会因为看错东西或记错步骤而出错。
Glossary
Neurosymbolic Agent (神经符号代理)
结合深度学习的感知能力与符号推理的规划机制，旨在实现高效、可靠的长远任务执行。
本文提出的机器人系统核心，即利用神经网络进行感知，符号模型进行推理。
PDDL (Planning Domain Definition Language, 规划域定义语言)
一种用于定义动作预条件、效果和任务目标的符号规划语言，广泛应用于自动规划研究。
本文中用PDDL描述动作模型，确保生成的计划在符号层面具有可行性。
Monte Carlo Tree Search (蒙特卡洛树搜索)
一种基于随机采样的搜索算法，用于长远规划中评估动作序列的价值。
在本文中，结合启发式评估，利用MCTS搜索最优的行动路径。
Vision-Language Model (视觉-语言模型)
能理解视觉信息与自然语言的深度学习模型，支持多模态任务理解。
如Qwen3.5，用于主动探索环境，获取目标相关的符号信息。
Symbolic State (符号状态)
用谓词和实例描述的环境抽象表示，便于符号推理和规划。
第一阶段探索后形成的符号状态作为第二阶段规划的基础。
Constrained Decoding (受约束解码)
在生成过程中动态限制输出内容，确保符合预定义的语义或结构规则。
本文中限制动作符号的生成，保证动作的预条件满足。
Long-Horizon Planning (长远规划)
考虑多步骤、多目标的行动序列，优化整体任务完成率。
通过MCTS和启发式评估实现长远目标的高效搜索。
Symbolic Reasoning (符号推理)
基于符号表示进行逻辑推导和决策的过程。
用于确保动作的可行性和任务的长远达成。
Transition Model (环境转移模型)
描述环境状态变化的数学模型，用于验证计划的可行性。
符号模型的正确性保证了计划在符号层面可行。
Perception-Planning Fusion (感知-规划融合)
结合感知信息与符号表示，实现环境状态的准确建模。
第一阶段的视觉探索与符号状态的结合。
Abstract
Language and vision-language models generate plausible embodied plans but do not guarantee executability, as their outputs can violate environment dynamics or act on incorrectly grounded entities. We present a neurosymbolic agent that factors long-horizon household tasks into task-directed visual exploration and constrained symbolic planning. In the first phase, a vision-language model and exploration harness acquire goal-relevant predicates and instance bindings from egocentric observations and grounded interactions, producing a symbolic initial state. In the second, a PDDL transition model restricts decoding to tokens that extend applicable actions. Monte Carlo tree search then evaluates executable continuations using a domain-independent planning heuristic. The resulting plans are executable by construction under the transition model, with transfer to the environment conditioned on correct visual grounding. On VirtualHome and ALFWorld, open 4B-27B models exceed 90% success in both environments, and our smallest agent substantially outperforms a 27B direct visual policy in each. Constraints and search prove complementary rather than interchangeable: in ALFWorld either alone solves under a third of tasks, whereas their combination solves over 95%. The method also uses several times fewer generated tokens than extended thinking and far fewer model-visible images than direct interaction, and residual failures localize to state acquisition rather than plan generation without any specialized training.
References (20)
XGrammar-2: Dynamic and Efficient Structured Generation Engine for Agentic LLMs
Linzhang Li, Yixin Dong, Guanjie Wang et al.
Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning
Saibo Geng, Martin Josifosky, Maxime Peyrard et al.
PDDL2.1: An Extension to PDDL for Expressing Temporal Planning Domains
M. Fox, D. Long
VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation
Naoki Yokoyama, Sehoon Ha, Dhruv Batra et al.
ReAct: Synergizing Reasoning and Acting in Language Models
Shunyu Yao, Jeffrey Zhao, Dian Yu et al.
End-to-end PDDL Planning with Hardcoded and Dynamic Agents
Emanuele La Malfa, Ping Zhu, Samuele G. Marro et al.
Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generation
Luca Beurer-Kellner, Marc Fischer, Martin T. Vechev
How Far Are LLMs from Symbolic Planners? An NLP-Based Perspective
Ma'ayan Armony, Albert Meroño-Peñuela, G. Canal
MiMo-Embodied: X-Embodied Foundation Model Technical Report
Xiaoshuai Hao, Lei Zhou, Zhijian Huang et al.
The FF Planning System: Fast Plan Generation Through Heuristic Search
J. Hoffmann, Bernhard Nebel
Mastering the game of Go without human knowledge
David Silver, Julian Schrittwieser, K. Simonyan et al.
PDDL-the planning domain definition language
D. McDermott, M. Ghallab, A. Howe et al.
Planning in the LLM Era: Building for Reliability and Efficiency
Michael Katz, Harsha Kokel, Kavitha Srinivas et al.
Landmarks, Critical Paths and Abstractions: What's the Difference Anyway?
M. Helmert, C. Domshlak
Bandit Based Monte-Carlo Planning
Levente Kocsis, Csaba Szepesvari
PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change
Karthik Valmeekam, Alberto Olmo, S. Sreedharan et al.
One Demo Is All It Takes: Planning Domain Derivation with LLMs from A Single Demonstration
Jinbang Huang, Yixin Xiao, Zhanguang Zhang et al.
Inner Monologue: Embodied Reasoning through Planning with Language Models
Wenlong Huang, F. Xia, Ted Xiao et al.
Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models
Yifu Yuan, Yao-Bang Huang, X. Yao et al.
UniGoal: Towards Universal Zero-shot Goal-oriented Navigation
Hang Yin, Xiuwei Xu, Lingqing Zhao et al.