Physical Agentic AI: An Architecture for Orchestrating a Robot Crew with LLMs
Introduced Physical Agentic AI framework to coordinate heterogeneous robot teams using LLMs, reducing error dispatch rates from 23%-29% to 0%.
Key Findings
Methodology
The Physical Agentic AI framework separates semantic planning from physical execution. It includes a non-actuating Mission Planner that generates task workflows based on robot skill libraries and a deterministic Robot Orchestrator that validates and authorizes each action step before execution.
Key Results
- Result 1: Skill retrieval improved task grounding from 51% to 96% in a drone-UGV search mission but still resulted in 23%-29% faulted dispatches.
- Result 2: Introducing the Orchestrator reduced faulted dispatch rates to 0% and intercepted all 8 injected faults.
- Result 3: Hardware tests using Unitree G1 and Go2 robots successfully completed two physical trials, validating the framework's real-world applicability.
Significance
This research addresses the critical gap between LLM-generated plans and their safe execution in multi-robot systems. By combining LLMs with robot skill libraries and a deterministic Orchestrator, it ensures reliable task execution, particularly in dynamic and collaborative scenarios.
Technical Contribution
Key contributions include: 1) a novel architecture separating LLM planning from execution; 2) skill-grounded task planning and validation mechanisms; 3) experimental validation across simulation and hardware, demonstrating applicability to heterogeneous robot teams.
Novelty
This is the first framework to integrate LLMs with robot skill libraries and workflow contracts, addressing the issue of unsafe or infeasible actions generated by LLMs. The deterministic Orchestrator ensures execution safety.
Limitations
- Limitation 1: The framework relies on predefined skill libraries, limiting its ability to handle tasks outside the library's scope.
- Limitation 2: The Mission Planner's performance is constrained by the reasoning capabilities of the LLM and the quality of context retrieval.
- Limitation 3: Hardware experiments were limited in scale, requiring further validation in larger robot teams.
Future Work
Future work includes expanding skill library coverage, improving the reasoning capabilities of the Mission Planner, and validating the framework in more complex multi-robot environments.
AI Executive Summary
Current LLM-based robot systems face challenges in bridging semantic task planning with safe physical execution, particularly in dynamic multi-robot scenarios. This paper introduces the Physical Agentic AI framework, which separates semantic reasoning from execution to ensure safety and reliability.
The framework features a non-actuating Mission Planner that generates task workflows based on robot skill libraries and workflow contracts, and a deterministic Robot Orchestrator that validates and authorizes each action step before execution. Experiments on a drone-UGV search mission and a humanoid-quadruped transportation task demonstrated the framework's effectiveness, reducing faulted dispatch rates from 23%-29% to 0%.
This research provides a robust solution for coordinating heterogeneous robot teams, addressing long-standing issues of unsafe or infeasible actions in LLM-generated plans. Future work will focus on scaling the framework to more complex environments and expanding its applicability to broader domains.
Deep Analysis
Background
LLMs have recently been applied to translate natural language instructions into robot actions. However, existing methods struggle with ensuring the feasibility and safety of LLM-generated plans, especially in multi-robot systems. Classical architectures like ATLANTIS and ROS provide task execution frameworks but lack mechanisms to validate LLM-generated plans.
Core Problem
Coordinating heterogeneous robot teams in dynamic environments requires robust task decomposition, skill allocation, and cross-robot synchronization. LLM-generated plans often include infeasible or unsafe actions, particularly when task information is incomplete or changes dynamically.
Innovation
The Physical Agentic AI framework introduces a novel separation of semantic planning and physical execution. Key innovations include a non-actuating Mission Planner, skill-grounded planning, workflow contracts for multi-robot coordination, and a deterministic Orchestrator for execution validation.
Methodology
- �� Mission Planner: Uses LLMs to interpret user requests and generate task workflows.
- �� Skill Libraries: Robots expose executable skills with parameters and preconditions.
- �� Workflow Contracts: Define synchronization points and dependencies for multi-robot tasks.
- �� Robot Orchestrator: Validates each task step against skill libraries and contracts before execution.
Experiments
Experiments included simulation and hardware tests. In simulation, a drone-UGV team performed search-and-dispatch tasks to evaluate fault interception. Hardware tests involved Unitree G1 and Go2 robots completing object transportation tasks, validating real-world applicability.
Results
Skill retrieval improved task grounding from 51% to 96%, but faulted dispatch rates remained at 23%-29%. The Orchestrator reduced faulted dispatch rates to 0% and intercepted all injected faults.
Applications
The framework is applicable to industrial automation, disaster response, and warehouse logistics, particularly in scenarios requiring multi-robot coordination.
Limitations & Outlook
The framework depends on predefined skill libraries, limiting its adaptability to novel tasks. Hardware experiments were small-scale, requiring further validation in larger, more dynamic environments.
Plain Language Accessible to non-experts
Imagine a factory where the designer plans tasks, machines perform them, and a quality inspector ensures safety. The Mission Planner is the designer, skill libraries are the list of machine capabilities, and the Robot Orchestrator is the inspector. Even if the designer makes a mistake, the inspector ensures only safe and feasible tasks are executed, guaranteeing a reliable outcome.
ELI14 Explained like you're 14
Think of it like a team game where each player has special skills—one can build, another can fight. The AI is the team captain. It plans who does what but can’t control them directly. Each player checks if they can actually do the task. If not, they say no! This way, even if the captain messes up, the team stays safe and wins.
Glossary
Large Language Model (LLM)
An AI model capable of understanding and generating natural language, often used for task planning and dialogue systems.
Used to interpret user requests and generate task workflows.
Skill Library
A collection of executable robot skills, including parameters and preconditions.
The Mission Planner uses the skill library to generate task plans.
Workflow Contract
Defines synchronization points, dependencies, and constraints for multi-robot tasks.
Used by the Orchestrator to validate task feasibility.
Robot Orchestrator
A deterministic module that validates and authorizes each task step, ensuring safety.
Intercepts infeasible or unsafe tasks before execution.
Mission Planner
A non-actuating module based on LLMs that interprets user requests and generates task workflows.
Proposes plans that are validated by the Orchestrator.
Open Questions Unanswered questions from this research
- 1 How can skill libraries be expanded to support more complex tasks?
- 2 How can the Mission Planner's reasoning be improved for dynamic environments?
Applications
Immediate Applications
Industrial Automation
Coordinate multiple robots in factories to improve efficiency and safety.
Disaster Response
Deploy heterogeneous robot teams for search and material transport in disaster zones.
Long-term Vision
Smart Cities
Enable coordinated multi-robot systems for traffic, logistics, and security management.
Abstract
Agentic AI frameworks interpret open-ended task goals and decompose them into multi-step plans. Richer information about embodiment-specific capabilities, physical preconditions, and cross-robot coordination improves grounding, but does not eliminate infeasible, mistimed, or unsafe physical actions. Physical robot crews therefore require an explicit architectural interface between semantic planning and execution, where every planned action is verified against robot capabilities, system state, and workflow constraints before actuation. This paper introduces Physical Agentic AI, a framework for skill-grounded robot agent orchestration, in which each robot exposes a typed library of executable skills while a foundation model planner decomposes a task into phases and assigns each phase to a robot-skill pair. A Robot Orchestration layer exposes the skill library, robot state, named locations, and workflow contracts to a non-actuating Mission Planner, while a deterministic Robot Orchestrator validates and authorizes one skill at a time. We evaluate on a drone-UGV search-and-dispatch mission, where every mission in every condition is executed live in Gazebo, and on a humanoid-quadruped transportation task using hardware-equivalent skill interfaces plus two physical trials on a Unitree G1 and Go2. Varying planner knowledge and runtime enforcement independently, we find that retrieval raises skill grounding from 51% to 96% yet leaves informed planners dispatching 23-29% of faulted steps. Per-dispatch enforcement reduces false dispatch to 0% with no false blocks, and a held-plan ablation confirms that the gate, not plan variation, is responsible. Live execution makes the difference physical: without enforcement all eight injected faults crossed the orchestration boundary and six produced robot motion; with enforcement all eight were refused before motion.