Fly0: Persistent Metric Anchoring for Zero-Shot Aerial Vision-Language Navigation
Fly0 introduces a structured semantic-geometric interface for zero-shot UAV navigation, achieving over 20% success rate improvement and halving localization error.
Key Findings
Methodology
Fly0 employs a three-stage pipeline: first, a multimodal large language model (MLLM) performs structured semantic grounding of natural language instructions, identifying target points, regions, relations, and confidence scores; second, depth data is used for uncertainty-aware 3D lifting, converting 2D grounded regions into spatially consistent 3D anchors, which are then transformed into relation-aware navigable goals; third, these persistent goals are stored in a memory module and used by a high-frequency LiDAR-based geometric planner. This decoupling of semantic reasoning and geometric control addresses issues like latency, occlusion, and viewpoint changes, enabling robust navigation even under partial visual contact.
Key Results
- In simulation environments (AerialVLN, OpenFly) and real-world benchmarks (Fly0-Real), Fly0 outperformed state-of-the-art zero-shot baselines, with success rates increasing by over 23%, and localization errors reduced by approximately 50%. The system maintained stable trajectories with fewer collisions and lower latency, demonstrating effective handling of occlusion and dynamic obstacles. The low-frequency semantic updates (0.5-1Hz) combined with high-frequency control (50-100Hz) resulted in a significant robustness boost, especially in unstructured outdoor scenes.
- Compared to end-to-end policies and models embedding the MLLM within the control loop, Fly0's modular interface provided clearer interpretability and better generalization, especially when visual contact was lost or instructions involved complex spatial relations.
- The experiments validated that the interface design—semantic grounding, uncertainty-aware 3D lifting, and relation-aware goal generation—was crucial for achieving these gains, rather than the control or perception modules alone.
Significance
This work addresses a fundamental challenge in UAV autonomous navigation: how to reliably convert open-vocabulary semantic instructions into persistent, actionable geometric goals under uncertainty. By decoupling semantic reasoning from control, Fly0 enhances robustness against occlusion, viewpoint shifts, and communication delays, paving the way for practical deployment of vision-language guided UAVs in complex, real-world environments. The approach also offers a scalable framework for integrating multimodal perception with safety-critical control, advancing both academic research and industry applications in autonomous aerial systems.
Technical Contribution
The paper introduces a structured semantic output schema with explicit uncertainty and relation tokens, enabling robust 3D goal lifting and relation-aware goal synthesis. It formalizes the semantic-to-geometric interface, decoupling low-frequency language reasoning from high-frequency control, and implements a persistent goal memory with adaptive re-grounding. The integration of uncertainty-aware depth aggregation, relation-based goal offsetting, and LiDAR-based collision checking constitutes a novel engineering solution that enhances system stability and interpretability. These innovations collectively push the boundary of modular autonomous navigation systems, especially in the context of zero-shot, open-vocabulary tasks.
Novelty
Fly0 is the first to explicitly decouple semantic reasoning from geometric control in UAV vision-language navigation, introducing a structured interface that manages uncertainty, spatial relations, and persistent goal representation. Unlike prior end-to-end or purely geometric methods, it emphasizes the importance of a semantic-geometric contract, enabling reliable goal anchoring even when visual cues are intermittent or ambiguous. This approach represents a significant advancement in the integration of multimodal perception and safety-critical control for aerial robots.
Limitations
- The system's performance heavily depends on the accuracy of depth data; noisy or missing depth measurements can impair goal lifting and relation generation.
- Handling highly dynamic environments with multiple moving obstacles or targets remains challenging, as the current relation models may not capture complex interactions.
- Real-time computational costs, especially for large-scale semantic grounding and relation processing, could limit deployment on resource-constrained UAVs.
Future Work
Future research will focus on integrating reinforcement learning to optimize the re-grounding and goal validation processes, improving adaptability in dynamic scenes. Extending the framework to multi-robot coordination and exploring end-to-end training of the semantic-geometric interface could further enhance robustness. Additionally, incorporating more advanced perception modules and real-time uncertainty estimation will be key to deploying Fly0 in more complex, real-world scenarios.
AI Executive Summary
Fly0 presents a novel approach to UAV vision-language navigation by establishing a structured semantic-geometric interface that effectively decouples low-frequency semantic reasoning from high-frequency geometric control. Traditional methods often entangle semantic understanding with control, leading to issues such as latency, target loss during occlusion, and poor generalization in unstructured environments. Fly0 addresses these challenges by leveraging a multimodal large language model (MLLM) to produce structured semantic outputs, including target points, regions, spatial relations, and confidence scores, at a low update rate (~0.5-1Hz). These semantic cues are then transformed into persistent 3D anchors through uncertainty-aware depth aggregation and relation-aware goal generation, enabling the UAV to maintain a stable goal even when visual contact is temporarily lost.
The system architecture integrates a LiDAR-based geometric planner operating at high frequency (50-100Hz), which continuously executes collision-free trajectories based on the persistent goal anchors. This decoupling allows the semantic reasoning process to focus on open-vocabulary understanding without compromising control responsiveness. Extensive experiments in simulation and real-world environments demonstrate that Fly0 outperforms existing zero-shot navigation baselines, with success rate improvements exceeding 23% and localization errors halved. The approach significantly reduces collision rates and improves system stability, especially in complex outdoor scenes with occlusion and dynamic obstacles.
This work advances UAV autonomy by emphasizing the importance of interface design—specifically, how semantic uncertainty and spatial relations are managed and integrated into geometric control. While challenges remain—such as depth noise sensitivity and dynamic scene complexity—the results suggest that robust, scalable vision-language navigation for UAVs is achievable through structured, decoupled system architectures. Future directions include reinforcement learning integration, multi-robot collaboration, and real-time uncertainty management, promising broader deployment in practical applications like search and rescue, environmental monitoring, and autonomous delivery.
Deep Dive
Abstract
Current Visual-Language Navigation (VLN) methodologies face a trade-off between semantic understanding and control precision. While Multimodal Large Language Models (MLLMs) offer superior reasoning, deploying them as low-level controllers leads to high latency, trajectory oscillations, and poor generalization due to weak geometric grounding. To address these limitations, we propose Fly0, a framework that decouples semantic reasoning from geometric planning. The proposed method operates through a three-stage pipeline: (1) an MLLM-driven module for grounding natural language instructions into 2D pixel coordinates; (2) a geometric projection module that utilizes depth data to localize targets in 3D space; and (3) a geometric planner that generates collision-free trajectories. This mechanism enables robust navigation even when visual contact is lost. By eliminating the need for continuous inference, Fly0 reduces computational overhead and improves system stability. Extensive experiments in simulation and real-world environments demonstrate that Fly0 outperforms state-of-the-art baselines, improving the Success Rate by over 20\% and reducing Navigation Error (NE) by approximately 50\% in unstructured environments. Our code is available at https://github.com/xuzhenxing1/Fly0.