A Semantic Autonomy Framework for VLM-Integrated Indoor Mobile Robots: Hybrid Deterministic Reasoning and Cross-Robot Adaptive Memory
Proposes a six-layer semantic autonomy framework combining hybrid deterministic-VLM reasoning and cross-robot memory transfer, enabling efficient indoor robot navigation.
Key Findings
Methodology
This work introduces a six-layer semantic autonomy stack integrating a seven-step parametric resolver that bypasses VLM in 88% of instructions, processing them in under 0.1ms without GPU. When ambiguity arises, VLM reasoning is invoked. A five-category semantic memory supports cross-session and cross-robot knowledge transfer, promoting learned preferences into deterministic rules via shared digests. Experiments on two custom differential-drive robots validate the approach, achieving 100% semantic transfer and decision accuracy, with latency reductions of over 103,000 times, all on Raspberry Pi 5 hardware without onboard GPU.
Key Results
- The system successfully handled 82 scenario-level decisions across three sessions, with 100% semantic transfer accuracy (33/33, 95% CI [0.894, 1.000]) and decision accuracy, demonstrating effective cross-session and cross-robot knowledge transfer. The fast resolver managed 88% of instructions in less than 0.1ms, significantly reducing reliance on VLM inference. Cross-robot preference transfer achieved 100% success, confirming the robustness of the memory framework. These results showcase the system’s high efficiency, scalability, and practical deployment potential.
Significance
This research addresses critical bottlenecks in deploying VLM-based semantic navigation on real robots, notably inference latency, session memory loss, and multi-robot knowledge sharing. By enabling rapid, deterministic resolution of routine instructions and persistent memory, it paves the way for scalable, intelligent indoor robotic systems capable of understanding natural language, adapting over time, and collaborating across multiple units. The approach significantly advances the state-of-the-art in edge robotics, offering a practical solution that requires no training data and runs on low-cost hardware, thus broadening the applicability of semantic navigation in industry and service domains.
Technical Contribution
The paper introduces a layered architecture that combines fast deterministic resolution with deep VLM reasoning, supported by a novel seven-step parametric cascade that leverages environment graph structure and learned preferences. It develops a five-category semantic memory system with explicit scope, enabling cross-session and cross-robot transfer via shared digests. This design drastically reduces inference latency and enhances scalability, demonstrating a new paradigm for semantic autonomy on edge devices without training or GPU acceleration. The experimental validation confirms the approach’s effectiveness and robustness in real-world multi-robot scenarios.
Novelty
This is the first work to implement a hybrid deterministic-VLM reasoning architecture with a lightweight, session-persistent memory system capable of cross-robot transfer on physical robots. It uniquely combines a fast, rule-based resolver with deep VLM reasoning only when necessary, achieving unprecedented latency reduction and transfer accuracy. The integration of a structured semantic memory with explicit scope and transfer mechanisms addresses longstanding challenges in practical semantic navigation, marking a significant step beyond existing simulation-focused or单机器人研究。
Limitations
- The system relies on predefined preferences and semantic annotations, which may limit adaptability in highly dynamic or unstructured environments. While latency is greatly reduced, complex ambiguous instructions still invoke VLM, incurring delays. Hardware constraints on edge devices restrict model complexity, and future work needs to optimize models further for energy efficiency and robustness in diverse real-world scenarios.
Future Work
Future research will focus on online preference learning, active exploration strategies, and expanding multi-robot collaboration scales. Enhancing robustness in highly dynamic or cluttered environments, integrating more advanced perception modules, and optimizing models for lower power consumption are key directions. Additionally, extending the framework to outdoor or large-scale settings and incorporating continual learning mechanisms will further advance semantic autonomy.
AI Executive Summary
Indoor mobile robots are increasingly vital in industries like manufacturing, logistics, and service sectors. Despite advances in navigation algorithms such as ROS 2 Navigation 2, these systems lack the ability to interpret natural language instructions that express intent rather than precise coordinates. Vision-Language Models (VLMs) have emerged as promising tools to bridge this gap, enabling robots to understand and reason about semantic cues. However, their high inference latency, session memory loss, and limited validation in real-world multi-robot setups hinder widespread deployment. To address these issues, this work proposes a comprehensive six-layer semantic autonomy framework that integrates hybrid deterministic and deep VLM reasoning, supported by a novel cross-robot memory transfer system.
The core innovation is a seven-step parametric resolver that handles 88% of routine instructions in under 0.1 milliseconds without invoking VLM, significantly reducing latency and computational load. When instructions are ambiguous, the system escalates to VLM reasoning, which is invoked selectively. The semantic memory system categorizes knowledge into environment, preferences, capabilities, and task history, enabling persistent, cross-session, and cross-robot learning. Preferences learned through VLM interactions on one robot are promoted into deterministic rules and transferred to other robots via shared digests, achieving 100% transfer accuracy.
Experimental validation on two custom-built differential-drive robots demonstrates the system’s effectiveness, with 82 decision scenarios across three sessions. Results show perfect semantic transfer and decision accuracy, with latency reductions exceeding 103,000 times compared to naive VLM invocation. The entire system operates on Raspberry Pi 5 hardware with no onboard GPU, requiring zero training data. This work significantly advances practical semantic navigation, offering scalable, efficient, and robust solutions for multi-robot indoor environments. Future efforts will focus on online preference adaptation, larger-scale multi-robot coordination, and robustness in complex, dynamic settings.
Deep Dive
Plain Language Accessible to non-experts
想象你在一家繁忙的厨房里做饭。厨师（机器人）需要根据你的指示完成任务，比如“帮我拿个苹果”或“准备一份沙拉”。传统厨师只知道具体的步骤，比如“去冰箱拿苹果”，但不知道你为什么要苹果。现在，厨房里有一个聪明的助手（VLM），能理解你说的话背后的意思，比如“我想吃点水果”。但这个助手很慢，每次都要花很长时间思考。为了让厨师更快反应，厨房设计了一个聪明的流程：平时厨师根据记忆和规则快速决定目标，只有遇到模糊或新指令时才请助手帮忙。这种方法让厨房的反应既快又智能，能理解你真正的意图，还能记住你平时喜欢的食材，下次不用再问。这个系统让厨房变得更高效，也更懂你的需求，就像机器人在家里或工厂里一样，能更好地理解和执行复杂指令。
ELI14 Explained like you're 14
想象你在学校里，有个聪明的朋友（机器人）帮你完成任务。有时候，你只需要告诉他“帮我拿个苹果”，他就知道去拿苹果，因为他以前知道你喜欢吃水果。有时候，你说“我想休息一下”，他就不知道怎么办。这时候，他会请一个超级聪明的老师（VLM）帮忙理解你的意思，但老师很慢，要花几秒钟甚至几分钟才能回答。为了让朋友更快反应，系统设计了一套聪明的规则：平时朋友根据记忆和习惯，能在几毫秒内决定目标，不用每次都找老师帮忙。只有遇到特别模糊或新奇的指令时，才会请老师帮忙确认。这样一来，朋友既聪明又快，能理解你的意图，还能记住你平时喜欢什么，下次不用再问。这个系统让机器人变得更聪明、更快，能在你需要时立刻帮上忙，就像在学校里帮你完成各种任务一样。
Abstract
Autonomous indoor mobile robots can navigate reliably to metric coordinates using established frameworks such as ROS 2 Navigation 2, yet they lack the ability to interpret natural language instructions that express intent rather than positions. Vision-Language Models offer the semantic reasoning required to bridge this gap, but their inference latency (2-9 seconds per decision on consumer hardware) and session-by-session amnesia limit practical deployment. This paper presents the Semantic Autonomy Stack, a six-layer reference framework for semantically autonomous indoor navigation, and validates a complete instance featuring hybrid deterministic-VLM reasoning and cross-robot adaptive memory on physical robots with off-the-shelf edge hardware. A seven-step parametric resolver handles 88% of instructions in under 0.1 milliseconds without invoking a language model, camera, or GPU; only genuinely ambiguous instructions escalate to VLM reasoning. A five-category semantic memory framework with explicit scope taxonomy (global environment knowledge, per-operator preferences, per-robot capabilities) enables cross-session learning and cross-robot knowledge transfer: preferences learned through VLM interactions on one robot are promoted to deterministic resolution and transferred to a second robot via a shared compiled digest, achieving a measured latency reduction of 103,000-fold. Experimental validation on two custom-built differential-drive robots across 82 scenario-level decisions and three sessions demonstrates 100% semantic transfer accuracy (33/33, 95% CI [0.894, 1.000]), 100% semantic resolution accuracy, and concurrent multi-robot operation feasibility - all on Raspberry Pi 5 platforms with no onboard GPU, requiring zero training data.