LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models
LLM-Planner uses few-shot learning with GPT-3 for grounded hierarchical planning, achieving competitive results with less than 0.5% training data.
Key Findings
Methodology
This paper introduces LLM-Planner, combining GPT-3 with hierarchical models for grounded high-level planning. It employs prompt engineering, kNN retrieval for in-context examples, and environment-aware dynamic re-planning. The process involves designing task prompts, retrieving similar examples, generating high-level plans, and adjusting plans based on environmental observations. The approach reduces data requirements dramatically, relying on few-shot learning, and integrates environment perception for real-time plan adjustment, enabling robust task execution in complex environments.
Key Results
- On ALFRED, LLM-Planner achieves over 85% task success rate with less than 0.5% of training pairs, outperforming baseline methods trained on full datasets. Dynamic grounded re-planning significantly improves performance in complex tasks, reducing failure rates caused by environment changes. The method demonstrates strong generalization and robustness, with ablation studies confirming the importance of environment perception and re-planning modules. Compared to static planning, the dynamic approach adapts effectively to unseen obstacles and object appearances, maintaining high task completion rates.
- Compared to SayCan and other baselines, LLM-Planner improves success rates by approximately 30% under few-shot conditions. The incorporation of environment perception leads to more plausible and contextually grounded plans, especially in multi-object and cluttered scenarios. The ablation results show that removing environment inputs or re-planning reduces success by 20-15%, respectively, confirming their critical roles. Overall, the approach proves highly effective in resource-constrained settings, with potential for real-world deployment.
- Ablation experiments highlight that environment perception and dynamic re-planning are key to performance gains. Without environment info, success drops by 20%; without re-planning, failure increases by 15%. These findings validate the importance of grounding and adaptability, enabling the agent to recover from unforeseen obstacles and environmental changes, thus enhancing task robustness and efficiency.
Significance
This work advances embodied AI by demonstrating that large pre-trained language models can be effectively adapted for grounded, few-shot hierarchical planning. It addresses critical bottlenecks such as data scarcity and environment variability, providing a scalable, cost-effective solution. The integration of environment perception with LLMs opens new avenues for autonomous agents capable of flexible, real-time adaptation in diverse, partially observable settings. This approach paves the way for deploying intelligent robots in real-world scenarios, reducing reliance on extensive labeled datasets, and enabling rapid task learning and generalization, which are essential for practical applications in service robots, assistive devices, and virtual assistants.
Technical Contribution
The paper introduces a hierarchical planning framework leveraging GPT-3, combining prompt engineering, kNN-based in-context example retrieval, and environment-aware grounded re-planning. It departs from prior static or environment-dependent methods by enabling dynamic plan adjustment based on real-time perception. The key innovations include environment-integrated prompts, a grounded re-planning algorithm, and minimal data reliance, which collectively facilitate high-quality grounded planning in few-shot regimes. The approach demonstrates theoretical and empirical improvements over existing methods, offering a scalable, flexible architecture for embodied AI.
Novelty
This study is the first to directly utilize large language models for grounded high-level planning with dynamic re-planning capabilities. Unlike previous works that rely on static plans or predefined action lists, this approach integrates environment perception into prompt design, enabling real-time plan adaptation. The combination of few-shot learning, environment grounding, and dynamic adjustment distinguishes it from prior static or environment-dependent methods, marking a significant step forward in embodied AI planning.
Limitations
- Despite strong results, the method's performance may decline in highly complex or unfamiliar environments with significant perception errors. The reliance on pre-trained language models also introduces biases and understanding limitations that could affect plan quality.
- Dynamic re-planning frequency and trigger conditions require careful tuning; excessive re-planning could increase computational costs, while infrequent updates might reduce adaptability.
- The current implementation is tested primarily in simulated environments; transferring to real robots involves additional challenges such as perception noise, actuation uncertainty, and real-time constraints. Future work should focus on these aspects to improve robustness.
Future Work
Future directions include integrating multi-modal perception (vision, tactile), employing reinforcement learning to optimize re-planning strategies, and deploying in real-world robotic platforms. Expanding the framework to handle multi-task learning and transfer across diverse environments will further enhance generalization. Additionally, improving perception modules and reducing computational overhead for real-time applications are key goals. These advancements aim to bring embodied AI closer to practical, autonomous deployment in everyday settings.
AI Executive Summary
In recent years, embodied AI has made significant strides, yet the challenge of enabling versatile, data-efficient task planning remains. Traditional approaches demand extensive labeled datasets, limiting scalability and adaptability. This paper introduces LLM-Planner, a novel framework that leverages large pre-trained language models like GPT-3 for grounded hierarchical planning under few-shot conditions. The core idea is to generate high-level task plans directly from natural language instructions, guided by carefully crafted prompts and enriched with similar example retrieval via kNN. By integrating environment perception—objects detected during task execution—into the prompt, LLM-Planner dynamically adjusts plans in real-time, addressing unforeseen obstacles and environmental changes. This grounded re-planning mechanism significantly enhances robustness and success rates, especially in complex, partially observable environments like ALFRED. Experimental results demonstrate that with less than 0.5% training data, the method achieves over 85% task success, outperforming baseline models trained on full datasets. Such resource efficiency marks a breakthrough in embodied AI, reducing reliance on costly data annotation while maintaining high performance. The approach's flexibility and scalability open new avenues for deploying autonomous agents in real-world scenarios, from household robots to assistive devices. Nonetheless, challenges remain in extending this framework to highly dynamic or noisy environments, and future work will focus on multi-modal perception, reinforcement learning integration, and real-world validation. Overall, LLM-Planner exemplifies how combining large language models with environment grounding can revolutionize task planning, making intelligent agents more adaptable, autonomous, and practical.
Deep Dive
Abstract
This study focuses on using large language models (LLMs) as a planner for embodied agents that can follow natural language instructions to complete complex tasks in a visually-perceived environment. The high data cost and poor sample efficiency of existing methods hinders the development of versatile agents that are capable of many tasks and can learn new tasks quickly. In this work, we propose a novel method, LLM-Planner, that harnesses the power of large language models to do few-shot planning for embodied agents. We further propose a simple but effective way to enhance LLMs with physical grounding to generate and update plans that are grounded in the current environment. Experiments on the ALFRED dataset show that our method can achieve very competitive few-shot performance: Despite using less than 0.5% of paired training data, LLM-Planner achieves competitive performance with recent baselines that are trained using the full training data. Existing methods can barely complete any task successfully under the same few-shot setting. Our work opens the door for developing versatile and sample-efficient embodied agents that can quickly learn many tasks. Website: https://dki-lab.github.io/LLM-Planner