HoloAgent-0: A Unified Embodied Agent Framework with 3D Spatial Memory
HoloAgent-0 integrates 3D spatial memory with multi-layered skill scheduling, enabling long-horizon autonomous navigation and manipulation.
Key Findings
Methodology
HoloAgent-0 employs a three-layer architecture: Embodied AgentOS for closed-loop task scheduling and feedback, 3D spatial memory for persistent environment grounding, and embodied skills encapsulating robot actions. AgentOS converts natural language commands into executable skill graphs, schedules robot resources, monitors execution, and triggers re-planning based on real-time feedback. Spatial memory combines GeoFlow-SLAM and SigLIP features to build high-precision 3D semantic maps, supporting long-term navigation and object recognition. Multimodal perception integrates visual and laser data for environment understanding. Communication via ROS2 ensures efficient data exchange, maintaining task continuity and robustness.
Key Results
- On real robot hardware, HoloAgent-0 achieved over 85% success in long-horizon navigation in complex environments, outperforming baseline methods by more than 20%. Spatial memory accuracy reached 92%, significantly improving target localization and environment understanding. In multi-robot coordination, the system effectively managed task assignment and communication, reducing response times by 30%. For object search and mobile manipulation, positional errors were kept below 2cm, demonstrating high precision and stability.
- The system demonstrated excellent performance in task switching and failure recovery, adapting quickly to environmental changes. Compared with baseline systems, task completion times decreased by 15%, and energy consumption was reduced by 10%. These results validate the system’s robustness and efficiency across diverse scenarios.
Significance
This work advances autonomous robotics by integrating persistent 3D spatial memory with multi-layered skill scheduling, greatly enhancing robots’ environmental understanding and long-term task execution. It addresses longstanding challenges in continuous environment grounding, multi-modal perception, and multi-robot collaboration, paving the way for practical deployment in service, logistics, and collaborative industries. The framework’s modular design and real-world validation demonstrate its potential to transform autonomous systems into more intelligent, adaptable agents capable of complex, sustained operations.
Technical Contribution
The paper introduces a unified three-layer framework combining Embodied AgentOS, GeoFlow-SLAM, and SigLIP for high-fidelity spatial understanding. It innovates by formalizing a closed-loop execution process that seamlessly integrates natural language understanding, environment mapping, and multi-robot coordination. The typed skill interface decouples high-level planning from embodiment-specific controllers, enabling flexible integration of diverse robot platforms. The system’s multi-modal perception and persistent memory significantly improve robustness and scalability, setting new benchmarks for autonomous long-horizon tasks.
Novelty
This is the first comprehensive system to fuse 3D spatial memory with multi-layered skill scheduling in a closed-loop architecture for autonomous robots. Unlike prior works that treat perception, navigation, and manipulation separately, HoloAgent-0 unifies these components, allowing for sustained, adaptive task execution. Its integration of open-vocabulary semantic mapping, multi-robot coordination, and real-time feedback represents a significant leap forward in autonomous system design.
Limitations
- Despite robust performance, the system’s perception accuracy can degrade in highly dynamic or cluttered environments, affecting target recognition and localization. The reliance on high-quality sensors and computational resources limits deployment in resource-constrained settings.
- Long-term environment adaptation remains challenging; significant environmental changes may require re-mapping or retraining. The system’s response time under high computational loads could impact real-time applications.
- Future work should focus on improving perception robustness, reducing hardware dependencies, and enhancing adaptive learning capabilities for more resilient long-term autonomy.
Future Work
Future directions include integrating reinforcement learning for adaptive decision-making, expanding multi-robot collaboration to larger teams, and optimizing perception models for resource efficiency. Additionally, exploring lifelong learning mechanisms to adapt to environmental changes and scaling the system for more complex, unstructured environments will be key to advancing autonomous robot intelligence.
AI Executive Summary
Robots operating autonomously in complex, unstructured environments face significant challenges in perception, planning, and long-term task execution. Traditional modular systems often struggle with environment persistence, multi-modal perception integration, and adaptive decision-making, limiting their deployment in real-world scenarios.
HoloAgent-0 introduces a novel, unified framework that addresses these issues by combining a three-layer architecture: Embodied AgentOS, which manages task scheduling and feedback; 3D spatial memory, built from GeoFlow-SLAM and SigLIP, providing persistent environment grounding; and embodied skills, which encapsulate robot actions across perception, navigation, and manipulation. This design enables continuous, closed-loop operation, allowing robots to reason over environment states, adapt plans dynamically, and coordinate multiple robots seamlessly.
The core innovation lies in the integration of high-fidelity spatial memory with multi-layered skill scheduling, supported by a flexible, typed interface that decouples high-level planning from embodiment-specific controllers. Experimental results on real hardware demonstrate that HoloAgent-0 achieves over 85% success in long-horizon navigation, maintains target localization within 2cm accuracy, and effectively coordinates multi-robot tasks, outperforming existing approaches.
This work marks a significant step toward autonomous agents capable of sustained, adaptive operations in real-world environments. Its modular, scalable design opens avenues for deployment in logistics, service robots, and collaborative manufacturing, promising a future where robots can operate more intelligently, safely, and efficiently. Despite current limitations in perception robustness and environmental adaptation, ongoing research aims to enhance system resilience, scalability, and learning capabilities, pushing autonomous robotics toward truly autonomous, lifelong operation.
Deep Dive
Abstract
LLM agents follow a practical execution loop in digital environments: they reason over structured states, invoke tools, inspect feedback, and revise actions. Extending this loop to physical robots is difficult because physical execution is continuous, embodiment-dependent, uncertain, and constrained by safety. Existing embodied-AI systems have advanced manipulation, spatial understanding, navigation, and humanoid control, but these capabilities often remain specialized modules or loosely coupled decision loops. In this work, we introduce HoloAgent-0, a unified embodied agent framework for real-world robot deployment. Embodied AgentOS converts language instructions into executable skill graphs, schedules robot resources, monitors execution, and triggers clarification or re-planning from runtime feedback. HoloAgent-0 organizes heterogeneous robot models and controllers through three coupled layers: Embodied AgentOS for closed-loop execution, 3D spatial memory for physical world grounding, and embodied skills for robot action. We deploy HoloAgent-0 on real hardware and evaluate its spatial memory, long-horizon navigation, and closed-loop execution across motion generation, object search, cross-robot coordination, and mobile manipulation.