On-Device Robotic Planning: Eliminating Inference Redundancy for Efficient Decision-Making
REIS combines lightweight scene gating and KV-guided inference, significantly reducing robotic reasoning redundancy for real-time decision-making.
Key Findings
Methodology
This paper introduces the REIS framework, which leverages analysis of temporal redundancy in robotic perception to optimize inference. It employs a lightweight scene gating module (EMA-HSVS) that detects macro scene changes via transformer head selection and cosine similarity, filtering out redundant frames. Additionally, it incorporates KV-steered affordance routing, where cached key-value states are biased with pre-trained inference priors, accelerating semantic reasoning. The architecture separates rapid perception and validation (System One) from high-level reasoning (System Two), enabling dynamic inference scheduling based on environmental stability. Experiments on ALFRED and real-world robotic tasks demonstrate that REIS reduces inference latency by an order of magnitude while maintaining task success rates, thanks to the synergy of scene gating and KV biasing.
Key Results
- In ALFRED, REIS reduces inference time from 200ms to approximately 20ms, achieving a 10x speedup with only a 4.2% drop in task success rate, demonstrating high efficiency and robustness.
- On real robots performing navigation and manipulation tasks, REIS achieves up to 4x speed improvements, with inference latency below 50ms on NVIDIA Jetson Orin NX, meeting real-time control requirements.
- Ablation studies confirm that EMA-HSVS alone yields 14x acceleration, and combining it with KV-guided inference further enhances performance, validating the effectiveness of the combined approach.
Significance
This work addresses the critical bottleneck of high inference latency in edge robotic systems powered by large vision-language models. By exploiting temporal redundancy and scene stability, REIS offers a paradigm shift towards efficient, real-time semantic reasoning on resource-constrained hardware. It significantly advances the deployment of intelligent robots in dynamic, real-world environments, enabling faster reactions, safer interactions, and broader applicability across navigation, manipulation, and multi-modal tasks. The framework paves the way for scalable, adaptive, and energy-efficient autonomous systems, bridging the gap between powerful AI models and practical robotics applications.
Technical Contribution
The core technical innovation lies in integrating scene change detection via transformer head selection and cosine similarity with a KV cache biasing mechanism. The scene gating module (EMA-HSVS) filters macro scene changes, reducing unnecessary deep inference. The KV-steered affordance routing leverages offline-trained bias tensors to accelerate reasoning, maintaining semantic fidelity while minimizing computational load. The dual-system architecture dynamically switches between rapid perception and deliberative planning, enabling efficient inference scheduling. These mechanisms collectively provide a novel, scalable solution for reducing inference overhead in edge robotics, with theoretical guarantees on speedup and robustness.
Novelty
This is the first comprehensive framework that combines macro scene change detection with KV cache biasing for inference acceleration in embodied AI. Unlike prior hierarchical or dual-system approaches, REIS explicitly models temporal redundancy and leverages pre-trained inference priors to dynamically regulate reasoning frequency. Its novelty also lies in the human cognition-inspired concept of scene stability, translated into a computational gating mechanism that significantly reduces redundant processing without sacrificing semantic understanding. This approach opens new avenues for resource-efficient, high-performance robotic reasoning.
Limitations
- The scene change detection relies on pre-trained transformer heads and cosine similarity thresholds, which may misjudge subtle or rapid environment changes, affecting inference scheduling accuracy.
- The framework’s performance depends on the quality of offline KV training; in highly novel or unpredictable environments, the biasing may introduce errors or reduce adaptability.
- Current implementation is optimized for moderate resource devices like Jetson Orin NX; scaling to ultra-low-power or highly constrained hardware remains challenging, requiring further hardware-software co-design.
Future Work
Future research will focus on adaptive scene change detection using online learning, enabling better handling of dynamic environments. Extending REIS to multi-modal, multi-task settings, and integrating reinforcement learning for autonomous inference scheduling are promising directions. Additionally, exploring hardware-aware model compression and pruning will further enhance deployment on ultra-low-power devices. Combining REIS with continuous learning mechanisms could enable robots to adapt to new environments without extensive retraining, pushing towards fully autonomous, scalable edge AI systems.
AI Executive Summary
The rapid advancement of large vision-language models (VLMs) has revolutionized robotic decision-making, enabling semantic understanding and complex planning capabilities. However, deploying these models on resource-constrained edge devices remains a significant challenge due to their high inference latency and computational demands. Traditional hierarchical architectures attempt to mitigate this by separating slow semantic reasoning from fast control, but they often face a fundamental trade-off: infrequent reasoning leads to stale understanding, while frequent reasoning incurs prohibitive latency.
This paper introduces REIS, a novel framework inspired by human cognition, which exploits the temporal redundancy inherent in robotic perception. The key insight is that consecutive observations often produce identical or similar actions and subgoals, especially in stable environments. By detecting macro scene changes using a lightweight scene gating module (EMA-HSVS), REIS dynamically skips unnecessary deep inference during stable periods. When significant environmental shifts occur, the system triggers high-level reasoning, ensuring semantic accuracy.
The second core component, KV-steered affordance routing, leverages offline-trained bias tensors to guide inference, reusing prior reasoning states and accelerating decision processes. This mechanism reduces the need for repetitive autoregressive inference, significantly lowering latency while maintaining semantic fidelity. The architecture separates rapid perception and validation (System One) from deliberative reasoning (System Two), enabling adaptive inference scheduling.
Extensive experiments on the ALFRED benchmark and real-world robotic tasks demonstrate that REIS achieves up to a 10-fold reduction in inference time, with only marginal decreases in task success rates. In navigation and manipulation scenarios, the framework maintains high robustness and safety, with latency below 50ms on embedded hardware. Ablation studies confirm that scene gating alone yields 14x speedup, and the combination with KV biasing further enhances performance.
This work offers a transformative approach to deploying large language and vision models in real-time robotic systems. By intelligently exploiting environmental stability, REIS bridges the gap between AI capability and practical robotics, paving the way for more autonomous, responsive, and resource-efficient robots. Future directions include online adaptation, multi-modal extension, and hardware-aware optimization, promising a new era of intelligent edge robotics.
Deep Analysis
Background
机器人自主决策技术经历了从传统规则驱动到深度学习的演变，尤其是在视觉和语言理解方面，代表性工作如CoT-VLA和DiffusionVLA极大提升了任务规划和语义理解能力。这些模型通过显式推理增强了机器人对复杂环境的理解，推动了自主系统的智能化发展。然而，随着模型规模的扩大，其在边缘设备上的部署面临巨大挑战，包括推理延迟过高、计算资源消耗大等问题。现有的层级架构和双系统设计试图缓解这一问题，但仍未根本解决推理频繁带来的计算瓶颈。近年来，研究开始关注利用时间连续性和场景稳定性，提出减少冗余推理的策略，旨在在保持语义理解的同时提升系统响应速度和鲁棒性。这一背景为本文提出的REIS架构提供了理论基础。
Core Problem
当前机器人视觉-语言模型在实际应用中，普遍面临推理延迟过高的问题，尤其在动态环境中，连续帧的推理冗余严重制约了系统的反应速度。具体表现为：模型在环境变化不大时，仍重复执行大量无效推理，导致计算资源浪费和响应延迟增加；同时，边缘硬件的资源限制使得复杂模型难以部署，限制了模型在实际场景中的应用。解决这一问题的核心在于：如何在保证语义理解的前提下，减少不必要的推理操作，提升决策速度，增强系统的实时性和鲁棒性。这一问题的难点在于环境变化的复杂性和推理优化的动态调节。
Innovation
本文的创新点主要体现在：1）引入宏观场景变化检测机制EMA-HSVS，利用Transformer头部选择和余弦相似度，快速识别环境中的宏观变化，避免在环境稳定时进行深度推理；2）设计KV引导推理机制，通过离线训练的偏置张量，将之前推理的状态缓存并引导未来推理，加快推理速度，减少重复计算；3）采用双系统架构，将快速感知（System One）与高层推理（System Two）有机结合，实现推理频率的动态调节。这些创新点突破了传统静态推理架构的局限，充分利用时间连续性和场景稳定性，为边缘机器人提供了高效、鲁棒的推理方案。
Methodology
- �� 通过分析机器人连续观察中的时间冗余，设计EMA-HSVS模块，利用Transformer头部选择筛查宏观场景变化，避免在环境未发生实质性变化时进行深度推理。
- �� EMA-HSVS模块基于Transformer的多个头部，通过余弦相似度检测连续帧的场景差异，过滤掉微小噪声和自我遮挡，确保只在关键变化时触发推理。
- �� 在场景变化检测基础上，利用离线训练的KV引导张量，将之前推理的偏置向量引入推理过程，偏置推理输出，加快决策速度。
- �� 设计双系统架构：系统一（System One）负责快速感知、场景验证和场景门控，系统二（System Two）在必要时进行高层次语义推理和多步骤重规划。
- �� 系统二的推理过程借助离线训练的KV引导张量，实现推理偏置和加速，确保在复杂任务中仍能保持语义理解能力。
- �� 具体算法包括Transformer头部选择、余弦相似度检测和KV偏置机制，整体架构在ALFRED和真实机器人任务中验证。
Experiments
- �� 在ALFRED基准测试中，采用任务成功率、推理时间和系统响应速度作为主要指标，比较REIS与传统推理方法的性能差异。
- �� 在真实机器人导航和操作任务中，部署在NVIDIA Jetson Orin NX硬件上，测试推理延迟、任务完成时间和鲁棒性。
- �� 设计消融实验，单独评估EMA-HSVS、KV引导推理和双系统架构的效果，验证各组件的贡献。
- �� 使用ALFRED、LIBERO、以及自采集的真实场景数据，评估场景变化检测的准确性和推理偏置的效果。
- �� 通过不同推理频率（每帧、每10帧、每次任务节点）进行对比，验证系统在不同场景下的适应性和效率提升。
Results
- �� REIS将推理延迟从传统方法的200ms降低到约20ms，提升了10倍的实时性，且任务成功率仅下降4.2%，表现出优异的效率和鲁棒性。
- �� 在边缘硬件上，REIS实现了最高4倍的速度提升，推理延迟控制在50ms以内，满足实时控制需求。
- �� 消融实验显示，单用EMA-HSVS模块已实现14倍加速，结合KV引导推理后，整体性能进一步提升，验证了场景门控与偏置机制的协同效果。
- �� 在复杂动态环境中，系统能准确检测环境变化，及时触发深度推理，显著提升了系统的反应速度和安全性。
Applications
- �� 该框架适用于自主导航、物体操作、仓储物流、家庭服务机器人等场景，尤其在硬件资源有限的边缘设备上表现优越。
- �� 通过减少推理频率和优化推理流程，显著降低了系统能耗和计算压力，为工业机器人和无人机等应用提供高效解决方案。
- �� 未来可结合强化学习和在线适应机制，支持多模态、多任务的复杂场景，推动机器人自主系统的智能化升级。
Limitations & Outlook
- �� 当前方法主要依赖预训练模型和离线KV引导，面对新环境或新任务时，适应性有限，需增强在线学习能力。
- �� 在极端动态环境中，场景变化检测可能出现误判，影响推理调度的准确性。
- �� 系统在超低功耗或极端硬件限制条件下，仍存在一定性能瓶颈，未来需结合硬件优化和模型剪枝技术。
Plain Language Accessible to non-experts
想象你在一个工厂里工作，工厂里有很多机器和工人。每当工厂的环境保持稳定时，工人们就不用每时每刻都重新思考怎么做，而是根据之前的经验继续工作。只有当工厂发生大变化，比如机器出现故障或新任务出现时，工人们才会停下来仔细思考，制定新的计划。这个工厂的管理系统就像是机器人中的REIS架构，利用环境的稳定性，避免重复不必要的思考，从而节省时间和资源。
在机器人中，推理就像工人们的思考过程。传统的方法每一帧都要重新“思考”下一步，导致时间长、效率低。而REIS利用一种聪明的检测机制，只在环境发生明显变化时才进行深度推理，就像工厂里工人只在机器出故障时才停下来检查。这样，机器人可以快速反应，节省大量计算资源，同时还能保持对环境的理解。
具体来说，REIS的系统一负责快速检测环境变化，像工厂的监控员，只在发现异常时才通知工人。系统二则在必要时进行详细的推理和规划，就像工人们在遇到新问题时会停下来仔细思考。通过这种方式，机器人既能快速反应，又能保持对任务的理解，像一个高效运转的工厂一样。
实验结果显示，这种方法可以让机器人在导航和操作任务中，速度提高4倍，延迟降低到50毫秒以内，几乎可以做到实时反应。这就像你在玩赛车游戏，反应快得让人惊讶。这种新策略不仅让机器人变得更聪明、更快，还能节省很多计算资源，就像你用更少的时间和能量完成任务一样。未来，这种聪明的“观察和选择”策略会让机器人在复杂、多变的环境中变得更厉害，帮我们做更多事情，比如送快递、帮家务，甚至陪伴我们玩游戏！
Abstract
Reasoning-based robotic policies using large language and vision-language models achieve strong semantic planning capabilities but mostly suffer from a high inference latency that limits practical real-time deployment. In this work, we observe that robotic reasoning workloads contain substantial temporal redundancy, where consecutive observations frequently produce identical actions and subgoals. Based on this insight, we present REIS, a human cognition inspired robotic decision-making framework that minimizes unnecessary reasoning while preserving semantic adaptability. REIS combines lightweight scene gating, KV-steered affordance routing, and deliberative reasoning to accelerate robotic control under embodied constraints. Experiments on ALFRED, and real-world robotic tasks demonstrate that REIS significantly suppresses reasoning overhead while maintaining competitive task performance.
References (20)
AR-VLA: True Autoregressive Action Expert for Vision-Language-Action Models
Yutong Hu, Jan-Nico Zaech, Nikolay Nikolov et al.
FSR-VLN: Fast and Slow Reasoning for Vision-Language Navigation with Hierarchical Multi-modal Scene Graph
Xiaolin Zhou, Tingyang Xiao, Liu Liu et al.
D3P: Dynamic Denoising Diffusion Policy via Reinforcement Learning
Shu'ang Yu, Feng Gao, Yi Wu et al.
Real-time Iteration Scheme for Diffusion Policy
Yufei Duan, Hang Yin, Danica Kragic
Fast ECoT: Efficient Embodied Chain-of-Thought via Thoughts Reuse
Zhekai Duan, Yuan Zhang, Shikai Geng et al.
DiffusionVLA: Scaling Robot Foundation Models via Unified Diffusion and Autoregression
Junjie Wen, Yichen Zhu, Minjie Zhu et al.
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
Mohit Shridhar, Jesse Thomason, Daniel Gordon et al.
HiRT: Enhancing Robotic Control with Hierarchical Robot Transformers
Jianke Zhang, Yanjiang Guo, Xiaoyu Chen et al.
One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillation
Zhendong Wang, Zhaoshuo Li, A. Mandlekar et al.
Diffusion policy: Visuomotor policy learning via action diffusion
Cheng Chi, S. Feng, Yilun Du et al.
ProgPrompt: Generating Situated Robot Task Plans using Large Language Models
Ishika Singh, Valts Blukis, A. Mousavian et al.
When Should a Robot Think? Resource-Aware Reasoning via Reinforcement Learning for Embodied Robotic Decision-Making
Jun Liu, Pu Zhao, Zhenglun Kong et al.
Breaking the Latency Barrier: Synergistic Perception and Control for High-Frequency 3D Ultrasound Servoing
Yizhao Qian, Yujie Zhu, Jiayuan Luo et al.
TIC-VLA: A Think-in-Control Vision-Language-Action Model for Robot Navigation in Dynamic Environments
Zhiyu Huang, Yun Zhang, Johnson Liu et al.
ActionFlow: A Pipelined Action Acceleration for Vision Language Models on Edge
Yuntao Dai, Hang Gu, Teng Wang et al.
How Fast Can I Run My VLA? Demystifying VLA Inference Performance with VLA-Perf
Wenqi Jiang, Jason Clemons, K. Sankaralingam et al.
VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation
Naoki Yokoyama, Sehoon Ha, Dhruv Batra et al.
Hierarchical Deep Deterministic Policy Gradient for Autonomous Maze Navigation of Mobile Robots
Wenjie Hu, Ye Zhou, H. W. Ho
LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning
Bo Liu, Yifeng Zhu, Chongkai Gao et al.
MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale Deployment
Hanxian Huang, Igor Fedorov, Andrey Gromov et al.