RecoverFly: A Failure-Aware Reinforcement Learning Post-Training Framework for Aerial Vision-Language Navigation
RecoverFly introduces a failure-aware reinforcement learning post-training framework, boosting success rate by 3.12-8.37% on UAV VLN tasks.
Key Findings
Methodology
RecoverFly employs token-level PPO with grammar-constrained autoregressive action generation, integrating dynamic failure replay, a two-stage long-tail scene curriculum, and reference-policy KL regularization. The framework maintains the pretrained textual action interface, optimizes actions through online feedback, and revisits unresolved failures to enhance corrective learning. The failure replay pool stores unsuccessful initializations, regenerates rollouts with current policy, and prioritizes unresolved cases. The scene curriculum adjusts sampling weights across two stages—initial proportional sampling and subsequent equal scene sampling—to address data imbalance. KL regularization anchors policy updates to the initial and final policies, ensuring stability. These components collectively improve sample efficiency, robustness, and generalization, validated by experiments on TravelUAV showing significant performance gains.
Key Results
- RecoverFly improves success rate from 47.96% to 56.33% on the full test set, with a 10.08 percentage point increase in hard scenarios (from 49.30% to 59.72%). Navigation error decreases by 19.23 meters, indicating more precise control. In unseen environments, success rate increases by 5.39 percentage points, and NE drops, demonstrating strong generalization. Ablation studies confirm the effectiveness of failure replay and scene curriculum, with component contributions detailed in the paper.
Significance
This work advances UAV autonomous navigation by effectively utilizing sparse feedback and addressing long-tail scene distribution issues. The failure-aware RL post-training significantly enhances model robustness and transferability, enabling UAVs to operate reliably in diverse, complex environments. It bridges the gap between supervised imitation and reinforcement learning, offering a practical pathway for deploying autonomous UAVs in real-world scenarios such as search-and-rescue, environmental monitoring, and urban navigation, where robustness and adaptability are critical.
Technical Contribution
The core innovations include: 1) token-level PPO tailored for grammar-constrained autoregressive actions, ensuring valid control commands; 2) dynamic failure replay that revisits unresolved failure cases, improving sample efficiency; 3) a two-stage scene curriculum balancing scene distribution to mitigate data bias; 4) reference-policy KL regularization to stabilize policy updates. These elements form a unified framework that enhances corrective learning, scene adaptation, and policy stability, surpassing traditional behavior cloning and single-stage RL approaches.
Novelty
This is the first comprehensive integration of failure-aware replay, scene long-tail curriculum, and policy regularization into an end-to-end UAV vision-language navigation framework. Unlike prior works relying solely on imitation learning or basic RL, RecoverFly effectively leverages sparse failure signals and balances scene diversity, enabling robust, generalizable policies. Its token-level optimization and staged regularization introduce new theoretical and practical avenues for autonomous aerial navigation.
Limitations
- The approach depends on pretrained models and may struggle in extreme environments with sensor noise or severe occlusions. The failure replay mechanism might introduce bias in highly imbalanced long-tail scenarios. Additionally, training costs are substantial, especially for large-scale, real-world deployment, necessitating further efficiency improvements.
Future Work
Future directions include integrating multi-modal perception for richer environment understanding, developing adaptive scene curricula based on difficulty metrics, and exploring meta-learning techniques for rapid adaptation. Efforts will also focus on reducing computational costs and extending the framework to real-time deployment in diverse operational settings, aiming for fully autonomous UAV systems capable of handling unpredictable scenarios.
AI Executive Summary
Unmanned aerial vehicle (UAV) vision-language navigation (VLN) has become a vital component for autonomous aerial systems, enabling UAVs to interpret natural language instructions and visual observations for complex tasks. Despite recent advances with end-to-end models like AerialVLA, these systems primarily rely on behavior cloning, which limits their ability to correct errors during real-time operation. Such policies are vulnerable to error accumulation, especially in long trajectories, occluded views, and rare scene configurations. Moreover, the highly imbalanced datasets used for training often bias models toward frequent scenes, impairing their performance in unseen environments.
Addressing these challenges, Wang et al. propose RecoverFly, a novel failure-aware reinforcement learning (RL) framework designed for post-training enhancement of UAV-VLA policies. The core idea is to leverage interactive feedback through token-level PPO optimization, which respects the grammar constraints of autoregressive actions. This allows the UAV to refine its control commands based on actual navigation outcomes. A key component is the dynamic failure replay mechanism, which stores unresolved failure cases and actively revisits them with the current policy, thus improving sample efficiency and corrective learning. To further enhance generalization, a two-stage long-tail scene curriculum adjusts the sampling distribution—from initial proportional sampling to uniform scene sampling—ensuring that rare but critical scenarios are adequately represented during training. Additionally, reference-policy KL regularization constrains policy updates, preventing catastrophic drift and preserving learned capabilities.
Experimental results on the TravelUAV benchmark demonstrate the effectiveness of RecoverFly. It achieves a success rate increase of 3.12 to 8.37 percentage points over the baseline AerialVLA, with notable improvements in long, unseen environments and rare scene configurations. The success rate on the full test set reaches 56.33%, significantly higher than prior methods, while navigation errors are reduced by nearly 20 meters. These findings validate the framework’s robustness, adaptability, and potential for real-world deployment. Overall, RecoverFly offers a comprehensive solution to key limitations in UAV-VLN, paving the way for more reliable, autonomous aerial navigation in complex, dynamic environments.
Deep Dive
Key Concepts
Token-level PPO
An optimization method that updates each token in autoregressive actions individually, ensuring grammar compliance and stable learning.
Failure replay
A mechanism to store and revisit unresolved failure cases, enhancing corrective learning from sparse feedback.
Long-tail scene curriculum
A training strategy that adjusts scene sampling to balance common and rare scenarios, improving generalization.
KL regularization
A technique to constrain policy updates by anchoring to reference policies, maintaining learned capabilities.
Autoregressive actions
Sequential control commands generated step-by-step, allowing complex, grammar-constrained flight behaviors.
Open Questions Unanswered questions from this research
- 1 如何在极端复杂环境中进一步提升模型的鲁棒性和泛化能力仍待探索，尤其是在传感器噪声和动态障碍条件下的表现。
- 2 现有方法在大规模实际应用中训练成本较高，未来需要研究更高效的训练策略和模型压缩技术。
Abstract
Unmanned aerial vehicle vision-language navigation (UAV-VLN) requires agents to translate visual observations and language instructions into reliable flight actions in complex environments. Although recent end-to-end UAV vision-language-action (UAV-VLA) policies reduce reliance on separately designed perception, planning, and control modules, their behavior-cloning objectives provide limited corrective supervision for interactive closed-loop execution. Reinforcement learning (RL) offers a promising solution, while its effectiveness is constrained by inefficient use of samples, long-tailed scene distributions, and policy distribution shift during optimization. To this end, we propose RecoverFly, a failure-aware RL post-training framework for end-to-end UAV-VLA policies. Specifically, RecoverFly adapts token-level RL for stable optimization of grammar-constrained autoregressive UAV actions, revisits unresolved failure cases to strengthen corrective learning and sample utilization, and combines a two-stage long-tail scene curriculum with reference-policy regularization to improve scene adaptation while preserving acquired capabilities. Experiments on the TravelUAV benchmark demonstrate that RecoverFly achieves the best performance on the seen, unseen-map, and unseen-object splits. Moreover, compared to the AerialVLA initialization, RecoverFly improves success rate by 3.12 to 8.37 percentage points under a total rollout budget of about 30\% of the training-set size, validating its effectiveness, robustness, and generalization capabilities.