Learning to Localize Reference Trajectories in Image-Space for Visual Navigation
LoTIS employs image-space localization of reference trajectories using Transformer architecture, achieving 94-98% success in diverse environments without robot-specific training.
Key Findings
Methodology
LoTIS utilizes frozen DINOv3 backbones for feature extraction, combined with a Transformer-based encoder-decoder framework. The offline trajectory encoder (ET) processes the entire reference path, generating spatio-temporal tokens. The online query encoder (Eq) extracts current view features, which are fused with trajectory features via cross-attention (FqT). A recurrent Transformer head iteratively predicts 2D image coordinates, visibility, and normalized distances for each trajectory point, trained with cross-trajectory sampling to enhance robustness against camera and viewpoint variations. The approach decouples perception from control, enabling robot-agnostic guidance.
Key Results
- In extensive simulation and real-world tests, LoTIS achieved success rates of 94-98% on forward navigation tasks, outperforming state-of-the-art methods like ViNT and PlaceNav by 20-50%. It demonstrated robust backward traversal, with success rates over 80%, and maintained high performance under camera intrinsics mismatch and viewpoint shifts. The model runs at approximately 22Hz on Jetson Orin AGX, confirming real-time applicability. Its ability to generalize across diverse environments and platforms was validated through experiments involving aerial robots and quadrupeds, showing significant improvement over baselines.
- Cross-trajectory training enabled the model to handle different camera parameters and viewpoints, leading to successful navigation from phone videos to multiple robot platforms. Ablation studies confirmed that joint trajectory processing and cross-attention mechanisms contributed to accuracy and robustness, especially in complex and dynamic scenes. The model's performance remained stable even when initialized far from the trajectory, highlighting its potential for long-range and reverse navigation tasks.
- The extensive dataset, comprising 25,000 trajectories and 850,000 query images across 500 environments, underpins the model's strong generalization. Real-world experiments confirmed its effectiveness in cluttered indoor, outdoor, and urban settings, with success rates exceeding 90%. The approach's simplicity and efficiency make it suitable for deployment in practical robotic systems, paving the way for scalable, platform-agnostic autonomous navigation.
Significance
LoTIS addresses fundamental limitations in visual navigation by removing the dependence on robot-specific training and camera calibration. Its image-space trajectory localization offers a universal interface compatible with various robot types and control schemes, facilitating rapid deployment in diverse scenarios. The method's robustness to viewpoint and camera variations significantly advances the field, enabling long-range, reverse, and multi-turn navigation in unstructured environments. This breakthrough has profound implications for autonomous systems in industry, search and rescue, and service robotics, where adaptability and reliability are crucial. Furthermore, its lightweight architecture supports real-time operation on embedded hardware, making it highly practical.
Technical Contribution
The paper introduces a Transformer-based architecture that jointly encodes entire reference trajectories and current views, employing cross-attention for robust matching. The cross-trajectory training strategy enhances generalization to unseen environments and camera parameters. The model outputs a comprehensive image-space representation—coordinates, visibility, and normalized distances—serving as an interface for classical local planners. This approach decouples perception from control, enabling robot-agnostic guidance and facilitating seamless integration with diverse platforms. The combination of offline trajectory encoding and online query matching constitutes a novel contribution, pushing the boundaries of visual navigation research.
Novelty
This work is the first to formulate trajectory localization purely in image space, independent of robot kinematics or camera calibration, enabling true cross-embodiment navigation. Unlike prior methods that rely on subgoal retrieval or end-to-end action prediction, LoTIS directly predicts the spatial position of trajectory points, supporting reverse and multi-turn navigation. Its cross-trajectory training strategy and Transformer architecture provide a new paradigm for robust, platform-agnostic visual navigation, addressing longstanding issues of viewpoint sensitivity and platform dependence.
Limitations
- The model's performance can degrade in highly dynamic or occluded environments, as it primarily relies on static feature matching. Extreme lighting or severe viewpoint deviations may reduce localization accuracy. Computational costs, while optimized for embedded hardware, still pose challenges for larger-scale or multi-robot deployments. Additionally, the approach assumes some degree of visual overlap with the reference trajectory, which may limit applicability in highly cluttered or sparse environments. Future work should focus on dynamic scene understanding and multi-modal integration to overcome these limitations.
Future Work
Future directions include integrating environmental perception modules such as depth or LiDAR for enhanced robustness, extending the framework for multi-robot collaboration, and exploring adaptive training strategies to handle highly dynamic scenes. Developing more efficient architectures to reduce computational load and improve scalability is also a priority. Additionally, incorporating higher-level semantic understanding could enable goal-aware navigation and task planning, broadening the system's applicability in real-world scenarios such as disaster response, autonomous delivery, and urban mobility.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在一个大型商场里迷路了，但你有一段朋友录制的路线视频。你只需看着视频，找到你当前所在的地方，然后沿着视频中的路线走。无论你站在商场的哪个角落，或者你用的是手机还是平板，只要能看到商场的景象，这个方法都能帮你找到正确的路径。它就像有一个神奇的地图助手，能在你看不到完整地图的情况下，告诉你下一步该走哪里。这个技术背后用的是一种聪明的算法，能在不同的视角和设备上都工作，帮助机器人在复杂环境中找到目标路线，就像你用手机导航一样简单。
ELI14 Explained like you're 14
想象你在一个超级大的游乐场里玩，想找到一条隐藏的路线，但你不知道具体在哪。以前，你得用地图或者记住每个转弯的细节，但现在，有一种神奇的工具，只要你拍一段视频，它就能告诉你怎么走，甚至帮你倒着走回起点！它就像一个超级聪明的朋友，能看着你现在的画面，告诉你距离目标有多远，在哪个方向。它不用提前知道游乐场的布局，也不用专门为你设计路线，只要你给它看一眼，它就能帮你找到正确的路径。这让你在玩游戏或者探索新地方时变得超级方便，机器人也能用这个技术在复杂环境中找到目标，真是太酷了！
Abstract
We present LoTIS, a model for visual navigation that provides robot-agnostic image-space guidance by localizing a reference RGB trajectory in the robot's current view, without requiring camera calibration, poses, or robot-specific training. Instead of predicting actions tied to specific robots, we predict the image-space coordinates of the reference trajectory as they would appear in the robot's current view. This creates robot-agnostic visual guidance that easily integrates with local planning. Consequently, our model's predictions provide guidance zero-shot across diverse embodiments. By decoupling perception from action and learning to localize trajectory points rather than imitate behavioral priors, we enable a cross-trajectory training strategy for robustness to viewpoint and camera changes. We outperform state-of-the-art methods by 20-50 percentage points in success rate on conventional forward navigation, achieving 94-98% success rate across diverse sim and real environments. Furthermore, we achieve over 5x improvements on challenging tasks where baselines fail, such as backward traversal. The system is straightforward to use: we show how even a video from a phone camera directly enables different robots to navigate to any point on the trajectory. Videos, demo, and code are available at https://finnbusch.com/lotis.