LiveVLN: Breaking the Stop-and-Go Loop in Vision-Language Navigation

Vision-language navigation (VLN) is a research field focused on enabling embodied agents to follow language instructions based on egocentric visual observations. Traditional VLN systems rely on strong cross-modal pretraining and Transformer-based reasoning, such as VLN-BERT and DUET, which have achieved strong benchmark performance on tasks like R2R and RxR. However, these systems still face challenges in continuous execution during real-world deployments, exhibiting noticeable stop-and-go behavior. This issue stems from a structural bottleneck rather than purely computational limitations. Most VLN systems still rely on a blocking three-stage interface consisting of sense, inference, and execution, causing even policies with strong benchmark performance to exhibit stop-and-go motion in streaming deployments.

The core problem faced by vision-language navigation systems in real-world deployments is the stop-and-go phenomenon. This issue arises from the blocking nature of the sense-inference-execution loop: after each new observation, the controller must wait for sensing, transmission, and inference to complete before motion can continue. This blocking leads to significant waiting times, limiting the system's real-time capabilities and continuity. Addressing this problem is crucial for enhancing the practical applicability of navigation systems, especially in scenarios requiring rapid response and continuous motion.

LiveVLN's core innovation lies in its training-free design and multi-step action continuation mechanism. • This mechanism allows refreshed future actions to be handed off before the current executable prefix is exhausted, reducing idle waiting and enhancing online execution smoothness. • Unlike traditional vision-language navigation systems, LiveVLN does not require retraining of pretrained models but achieves continuous execution through a runtime framework. • This innovation offers a new approach to solving the stop-and-go issue, potentially impacting both academia and industry in navigation technology development.

The implementation of LiveVLN involves the following key steps: • Decoupling the current execution stage from the next sense and inference stages through a short-horizon action state. This state comprises executed actions, a guard buffer, and a revisable tail. • Operating at runtime, allowing integration with compatible pretrained vision-language model navigators. • Utilizing a multi-step action continuation mechanism, allowing refreshed future actions to be handed off before the current executable prefix is exhausted. • In this way, only the minimal prefix necessary to sustain continuous motion is committed, leaving later actions open to revision based on newer observations.

The experimental design includes evaluating LiveVLN's performance on R2R and RxR benchmarks while measuring its continuity and wall-clock efficiency in real-world deployments. • Comparisons are made using the same checkpoints and deployment settings to ensure result comparability. • Evaluation metrics include waiting time, waiting ratio, visible gap, pause count, and wall-clock episode duration to examine whether the runtime hides sense-and-inference latency. • In the real-robot study, both navigators are deployed on the same Unitree G1 client-server platform, with Wi-Fi jitter remaining within controllable limits.

Experimental results show that LiveVLN reduces waiting time by more than 50% on both StreamVLN and NaVIDA, markedly reduces pause count, and shortens wall-clock episode time by 12.6% to 19.6%. • LiveVLN preserves benchmark performance on R2R and RxR while reducing waiting time and improving action availability. • Ablation studies reveal that removing the revisable tail or real-time adaptation weakens performance, indicating that these components are crucial for maintaining task success and hiding latency.

LiveVLN's application scenarios include navigation tasks requiring rapid response and continuous motion, such as robotic delivery, autonomous driving, and drone navigation. • These applications require efficient real-time navigation systems to handle complex environments and dynamic changes. • LiveVLN provides a more efficient solution for these applications by reducing waiting time and increasing action availability.

Despite LiveVLN's impressive performance in reducing waiting time and increasing action availability, it may not completely eliminate stop-and-go behavior in some cases, especially in environments with high latency and communication jitter. • Physical execution still dominates, thus limiting overall efficiency gains. • The framework's adaptability relies on accurate latency estimation, which may perform poorly in scenarios with significant latency variation. Future research directions include further optimizing LiveVLN's real-time adaptability to handle more complex environments and larger latency variations.