NavTrust: Benchmarking Trustworthiness for Embodied Navigation

Embodied navigation refers to the ability of robots to autonomously move in complex environments, often relying on vision and language instructions. In recent years, significant progress has been made in the field of embodied navigation with the development of deep learning and computer vision technologies. However, existing navigation systems perform poorly when faced with common real-world perception and language input corruptions. For example, Vision-Language Navigation (VLN) and Object-Goal Navigation (OGN) experience significant performance drops when encountering minor linguistic perturbations or small domain shifts. These vulnerabilities are often overlooked in existing benchmarks, which typically report performance under idealized input conditions. Additionally, current benchmarks lack a unified framework for systematically evaluating robustness mitigation strategies. To address these gaps, NavTrust provides a unified benchmark that systematically introduces RGB, depth, and instruction corruptions in realistic scenarios to evaluate the performance of embodied navigation models.

Existing embodied navigation systems perform poorly when faced with common real-world perception and language input corruptions. These corruptions include RGB image blurring, low lighting, noise, depth sensor Gaussian noise, data loss, multipath interference, quantization errors, and language instruction variations. These corruptions can lead to significant performance degradation in navigation systems, affecting their applicability in real-world scenarios. Therefore, evaluating and improving the robustness of embodied navigation systems under these corruption conditions is an important and challenging problem.

NavTrust's innovations lie in its provision of a unified framework for evaluating the robustness of embodied navigation systems under various input corruptions. Specifically:

• �� NavTrust systematically introduces RGB, depth, and instruction corruptions to evaluate the performance of embodied navigation models. These corruptions include RGB image blurring, low lighting, noise, depth sensor Gaussian noise, data loss, multipath interference, quantization errors, and language instruction variations.

• �� NavTrust is the first to expose embodied navigation agents to diverse RGB-depth corruptions and instruction variations within a single framework.

• �� NavTrust provides evaluations of multiple mitigation strategies, including data augmentation, knowledge distillation, adapter tuning, and large language model fine-tuning, offering an empirical roadmap for future robustness enhancements.

The research methodology of NavTrust includes the following key steps:

• �� Dataset Selection: Use the validation set from the Habitat-Matterport3D dataset for OGN evaluation; use the R2R and RxR datasets for VLN evaluation.

• �� Corruption Types: Introduce eight types of RGB image corruptions (e.g., motion blur, low lighting, noise) and four types of depth corruptions (e.g., Gaussian noise, data loss), as well as five types of instruction corruptions (e.g., masking, stylistic variation).

• �� Model Evaluation: Evaluate seven state-of-the-art methods, including ETPNav, NaVid, Uni-NaVid, WMNav, L3MVN, PSL, and VLFM.

• �� Mitigation Strategies: Evaluate four mitigation strategies, including data augmentation, knowledge distillation, adapter tuning, and large language model fine-tuning.

• �� Experimental Design: Conduct experiments in both simulated environments and real robots to evaluate model performance under different corruption conditions.

The experimental design includes the following aspects:

• �� Datasets: Use the validation set from the Habitat-Matterport3D dataset for OGN evaluation; use the R2R and RxR datasets for VLN evaluation.

• �� Baselines: Evaluate seven state-of-the-art methods, including ETPNav, NaVid, Uni-NaVid, WMNav, L3MVN, PSL, and VLFM.

• �� Evaluation Metrics: Use success rate (SR), success-weighted path length (SPL), and performance retention score (PRS) as metrics to evaluate model performance.

• �� Hyperparameters: Set corruption intensity to 0.5 to induce significant but realistic performance degradation.

• �� Ablation Studies: Evaluate the impact of different corruption types and mitigation strategies on model performance.

The experimental results show that:

• �� Under RGB image corruptions, RGB-only agents (e.g., Uni-NaVid and NaVid) are penalized more heavily than depth-involved or language-conditioned methods. Black-out and foreign-object corruptions reduce the success rate of RGB-only agents by 22% and 13%, respectively.

• �� Under depth corruptions, Gaussian noise is the most destructive: L3MVN's success rate collapses from 50% to 2%, and VLFM similarly drops from 50% to 0%.

• �� Under instruction corruptions, ETPNav, NaVid, and Uni-NaVid experience success rate declines of 28%, 12%, and 21%, respectively, under random masking.

• �� Data augmentation strategies improve model performance under corruption conditions to varying degrees, with per-episode data augmentation performing better under RGB and depth corruptions.

The findings from NavTrust have potential applications in several fields:

• �� Autonomous Driving: Improve the robustness of autonomous driving systems under adverse weather and lighting conditions, enhancing their navigation capabilities in complex environments.

• �� Service Robots: Enhance the navigation capabilities of service robots in home and commercial environments, particularly in complex and dynamic settings.

• �� Drone Navigation: Improve drone navigation performance in complex terrains and variable environments, supporting more application scenarios such as agricultural monitoring and disaster relief.

The NavTrust study has the following limitations:

• �� The evaluation primarily focuses on simulated environments, and although deployed on a real robot, further validation in more real-world scenarios is needed.

• �� The effectiveness of mitigation strategies varies across different models and corruption types, potentially requiring customization for specific applications.

• �� The current benchmark does not fully cover all possible perception and language input corruption types, which may need to be expanded in the future.