LocalNav: Distilling Frontier VLMs and Embodied RL for On-Device Object Goal Navigation
LocalNav distills frontier VLMs and RL to enable ObjectNav on embedded devices, reducing inference latency by 82.8%.
Key Findings
Methodology
This study introduces LocalNav, which distills reasoning capabilities from cloud VLMs (e.g., Claude Sonnet 4.6) into a lightweight Qwen3.5-4B model, combined with E-RLVR for token length optimization on embedded GPUs.
Key Results
- Result 1: Claude Sonnet 4.6 achieves 39.7% SR on HM3D OVON, while distilled Qwen3.5-4B achieves 34.5%, reducing the performance gap to 5.2%.
- Result 2: E-RLVR reduces token generation length by 72.1% and inference latency by 71.8%, with quantization achieving a total latency reduction of 82.8%.
- Result 3: Fine-tuning Qwen3.5-4B with 500 distillation traces significantly enhances navigation capabilities without requiring large-scale datasets.
Significance
This work addresses the latency and resource bottlenecks of deploying VLMs on embedded devices, enabling efficient ObjectNav for resource-constrained mobile robots and advancing edge computing in robotics.
Technical Contribution
Introduces E-RLVR with token generation regularization for inference efficiency; leverages scene graphs (SG) and Qwen3.5-4B for modular high-level decision-making, reducing reliance on cloud computation.
Novelty
First to distill spatial-semantic reasoning from large VLMs into an embedded-friendly model, combining E-RLVR and quantization for efficient local ObjectNav.
Limitations
- Limitation 1: Performance still lags behind cloud VLMs, especially in complex scenarios.
- Limitation 2: E-RLVR requires simulation environments, limiting real-world applicability.
- Limitation 3: Quantization may lead to minor inference accuracy loss.
Future Work
Future work includes exploring more efficient distillation methods, optimizing for multi-task scenarios, and validating long-term performance on real robots.
AI Executive Summary
Vision-language models (VLMs) have shown exceptional capabilities in open-vocabulary tasks like ObjectNav, but their high computational demands limit local deployment. LocalNav addresses this by distilling cloud VLM reasoning into a lightweight Qwen3.5-4B model and optimizing token generation via reinforcement learning (E-RLVR).
Built on Claude Sonnet 4.6's scene graph (SG) architecture, LocalNav fine-tunes Qwen3.5-4B using just 500 distillation traces, achieving a 34.5% success rate compared to the cloud model's 39.7%. E-RLVR reduces token generation length by 72.1% and inference latency by 71.8%, with quantization achieving an overall latency reduction of 82.8%.
This research provides an efficient ObjectNav solution for resource-constrained robots, showcasing the potential of edge computing in robotics. However, performance gaps remain, and future work will focus on further optimization and real-world validation.
Deep Analysis
Background
Vision-language models (VLMs) have revolutionized open-vocabulary tasks, particularly ObjectNav. However, their reliance on cloud computation introduces latency and network dependency, limiting their use in mobile robotics.
Core Problem
Existing VLM-based ObjectNav systems depend on cloud resources, leading to high latency and limited autonomy. Achieving efficient local inference on embedded devices remains a key challenge.
Innovation
Key innovations include:
1. A modular SG-based architecture separating high-level reasoning from low-level control.
2. Distilling cloud VLM reasoning into Qwen3.5-4B for local deployment.
3. Introducing E-RLVR with token generation regularization to optimize inference efficiency.
Methodology
- �� Distill reasoning from Claude Sonnet 4.6 into Qwen3.5-4B using 500 traces.
- �� Use scene graphs (SG) for environment representation, combining text and image prompts for high-level decisions.
- �� Apply E-RLVR to optimize token generation length and reduce inference latency, combined with quantization for further efficiency.
Experiments
Experiments on the HM3D OVON benchmark evaluated Claude Sonnet 4.6 and Qwen3.5-4B. Distillation and E-RLVR significantly improved Qwen3.5-4B's success rate and inference efficiency.
Results
Claude Sonnet 4.6 achieved 39.7% SR on HM3D OVON, while Qwen3.5-4B reached 34.5% after distillation and optimization. E-RLVR reduced token length by 72.1%, cutting latency by 71.8%.
Applications
This approach is ideal for resource-constrained mobile robots in tasks like household assistance and warehouse logistics.
Limitations & Outlook
The model's performance in complex scenarios is still below cloud VLMs, and E-RLVR's reliance on simulation environments may limit real-world deployment.
Plain Language Accessible to non-experts
Imagine LocalNav as a 'smart guide dog.' A guide dog understands commands (language), observes the environment (vision), and navigates to the target. LocalNav learns from 'cloud mentors' and runs efficiently on small devices, helping robots complete complex navigation tasks.
ELI14 Explained like you're 14
Think of LocalNav like a treasure hunt assistant in a video game! It can read maps (vision), understand your hints (language), and plan routes (navigation). It's super smart because it learned tricks from a 'cloud teacher' and now works on your small gaming console to help you win the game!
Glossary
Vision-Language Model (VLM)
A deep learning model combining vision and language to understand images and text.
Used for semantic reasoning in ObjectNav.
Object Goal Navigation (ObjectNav)
A task where robots locate specific objects based on instructions.
The primary task addressed in the paper.
Scene Graph (SG)
A graph-based representation of objects and their relationships in an environment.
Core to high-level reasoning in LocalNav.
Distillation
A technique to transfer knowledge from a large model to a smaller one.
Used to adapt cloud VLM capabilities to Qwen3.5-4B.
E-RLVR
A reinforcement learning method optimizing token generation length.
Improves inference efficiency in LocalNav.
Open Questions Unanswered questions from this research
- 1 How can the performance gap between local and cloud models be further minimized?
- 2 What are the long-term effects of E-RLVR in real-world deployments?
Applications
Immediate Applications
Home Service Robots
Helps robots locate specific items in homes, e.g., 'drawers below the microwave.'
Warehouse Robots
Optimizes item retrieval and transport efficiency in warehouses.
Long-term Vision
Fully Autonomous Navigation Robots
Enables complex tasks like disaster rescue and exploration without cloud dependency.
Abstract
Vision Language Models (VLMs) have emerged in the robotic domain as a powerful tool that enables environmental perception with language context, serving as a catalyst for open-vocabulary tasks like ObjectNav. Yet, their computational footprint typically confines them to cloud execution, hindering low-latency inference with local deployment on resource-constrained robots. To address this challenge, we present a distillation strategy that transfers complex spatial-semantic reasoning from large frontier models into a lightweight, 4B-parameter local VLM for edge execution on embedded GPU devices (e.g., Jetson Orin). We first establish a State of the Art (SotA), Scene Graph (SG)-based pipeline using Claude Sonnet 4.6, achieving a 39.7% Success Rate (SR) on the HM3D OVON benchmark. We then demonstrate that fine-tuning Qwen3.5-4B on just 500 frontier reasoning traces effectively enables navigation capabilities, yielding a SR of 34.5%, narrowing the gap to the performance of large cloud models. Finally, we introduce E-RLVR with Token Generation (TG) regularization to compress output sequence lengths for physical deployment while grounding the agent in its task. This downstream optimization reduces TG overhead by 72.1% and latency by 71.8%. Combined with quantization, this joint strategy yields a cumulative 82.8% reduction in overall inference latency without significantly sacrificing performance, presenting a viable paradigm for local, low-latency VLM execution on mobile robots.