Think at 5 Hz, Act at 20 Hz: Asynchronous Fast-Slow Vision-Language-Action Inference for Closed-Loop Driving
Proposes asynchronous fast-slow vision-language-action inference, boosting route completion from 37.0 to 94.0.
Key Findings
Methodology
The study introduces an asynchronous fast-slow architecture using a frozen 7B vision-language backbone as the slow system to process navigation instructions and visual history. A lightweight action expert acts as the fast system, accessing the cache and current camera frame at each simulation tick to regress waypoints. The expert is trained under randomized staleness to align training with asynchronous execution.
Key Results
- In CARLA's LangAuto-Short routes, the system produces fresh control at every 50 ms simulation tick, lifting route completion from 37.0 to 94.0.
- In a frame-skip ablation with the same expert, the expert alone raises the driving score, while per-tick freshness raises completion from 82.1 to 94.0 and cuts red-light violations by a third.
- Trained on a single town, the expert transfers zero-shot to two unseen towns, maintaining 84-94% route completion where the baseline reaches 31-41%.
Significance
This study addresses the inference latency problem of large language models in autonomous driving, significantly improving route completion and reducing traffic violations. The proposed architecture offers a novel solution for real-time control, particularly in complex driving scenarios.
Technical Contribution
The study's technical contribution lies in proposing an asynchronous fast-slow architecture that separates inference from real-time control, achieving low-latency control outputs through a caching mechanism. This method significantly enhances system efficiency and performance without compressing or retraining the backbone network.
Novelty
This study is the first to propose an asynchronous fast-slow architecture that separates scene understanding from real-time action inference, addressing the mismatch between inference latency and control rate in traditional methods.
Limitations
- The method lacks transferability in negotiating hazards on long routes, potentially requiring longer training data.
- The system is tested in a simulated environment, and its safety on real roads is yet to be validated.
Future Work
Future work could include testing the architecture in more complex traffic scenarios, exploring the impact of long-route training data, and applying it in real-world environments.
AI Executive Summary
The rapid development of autonomous driving technology faces the challenge of making real-time decisions in complex environments. Existing large language models excel in scene understanding and instruction execution but suffer from inference latency that mismatches the control rate required by vehicles, leading to some control outputs ignoring the latest observations.
This paper proposes an asynchronous fast-slow architecture that separates vision-language model inference from real-time control, addressing this issue. Specifically, a frozen 7B vision-language backbone acts as the slow system to process navigation instructions and visual history, while a lightweight action expert acts as the fast system, accessing the cache and current camera frame at each simulation tick to regress waypoints. The expert is trained under randomized staleness to align training with asynchronous execution.
Experimental results show that the system significantly improves route completion and reduces traffic violations in the CARLA simulator. This study offers a novel solution for real-time control in autonomous driving, particularly in complex driving scenarios. However, the method lacks transferability in negotiating hazards on long routes, and future work could include testing the architecture in more complex traffic scenarios.
Deep Analysis
Background
Autonomous driving technology has made significant progress in recent years, particularly with the application of vision-language models. These models can understand natural language instructions and explain their decision-making processes. However, existing methods face inference latency issues in real-time control, leading to some control outputs ignoring the latest observations.
Core Problem
The core problem is how to achieve real-time control while maintaining efficient scene understanding. The inference latency of large language models mismatches the control rate required by vehicles, leading to some control outputs ignoring the latest observations.
Innovation
This paper proposes an asynchronous fast-slow architecture that separates vision-language model inference from real-time control, addressing the mismatch between inference latency and control rate. The architecture uses a frozen 7B vision-language backbone as the slow system and a lightweight action expert as the fast system.
Methodology
- �� Use a frozen 7B vision-language backbone as the slow system to process navigation instructions and visual history.
- �� A lightweight action expert acts as the fast system, accessing the cache and current camera frame at each simulation tick to regress waypoints.
- �� Train the expert under randomized staleness to align training with asynchronous execution.
Experiments
Experiments were conducted on CARLA simulator's LangAuto-Short routes using a frozen 7B vision-language backbone and a lightweight action expert. The expert is trained under randomized staleness to align training with asynchronous execution. Results show the system produces fresh control at every 50 ms simulation tick, lifting route completion from 37.0 to 94.0.
Results
Experimental results show the system significantly improves route completion and reduces traffic violations in the CARLA simulator. The expert, trained on a single town, transfers zero-shot to two unseen towns, maintaining 84-94% route completion where the baseline reaches 31-41%.
Applications
The architecture can be applied in real-time control scenarios in autonomous driving, particularly in complex urban traffic environments. By reducing inference latency and improving control accuracy, the method is expected to enhance the safety and efficiency of autonomous driving systems.
Limitations & Outlook
The method lacks transferability in negotiating hazards on long routes, potentially requiring longer training data. The system is tested in a simulated environment, and its safety on real roads is yet to be validated.
Plain Language Accessible to non-experts
Imagine you're in a kitchen where the slow system is like a chef who understands recipes and prepares ingredients, while the fast system is like an assistant who quickly executes the chef's instructions. The chef may take time to understand complex recipes, but the assistant can act quickly at each step, ensuring the dish is completed on time. This way, the kitchen operates efficiently without sacrificing quality.
ELI14 Explained like you're 14
Imagine you're playing a racing game, and your task is to drive a car through a city. The game has a smart assistant that understands the route instructions you give and helps you make quick decisions on turns and brakes. This assistant is like a super brain in your game, ensuring you don't miss any traffic lights. This method makes you perform better in the game, just like the system in this paper excels in autonomous driving!
Glossary
Vision-Language Model
A model that combines visual and language information for reasoning, typically used for understanding complex scenes and instructions.
Used in this paper to process navigation instructions and visual history.
Asynchronous Architecture
An architecture that separates tasks and executes them at different times to improve system efficiency.
Used to separate vision-language model inference from real-time control.
CARLA Simulator
An open-source urban driving simulator used for autonomous driving research, providing complex traffic environments.
Used to test the system's route completion and traffic violations.
Randomized Staleness Training
A training method that introduces random delays to simulate system staleness in real-world applications.
Used to train the action expert, ensuring alignment with asynchronous execution.
Waypoint Regression
A technique for predicting a vehicle's future path based on current environment and historical information.
Used by the action expert to generate fresh control at each simulation tick.
Open Questions Unanswered questions from this research
- 1 How to maintain efficient real-time control in complex traffic scenarios on long routes?
- 2 What is the safety and reliability of this architecture in real-world road environments?
Applications
Immediate Applications
Urban Autonomous Driving
By reducing inference latency and improving control accuracy, this method is expected to enhance the safety and efficiency of autonomous driving systems in complex urban traffic environments.
Long-term Vision
Fully Autonomous Driving
The architecture has the potential to become part of a fully autonomous driving system, achieving fully autonomous urban driving through further optimization and validation.
Abstract
Large language models bring instruction following and scene reasoning to end-to-end driving, but their inference latency collides with the control rate a vehicle requires. Existing closed-loop agents hide this gap by invoking the model on alternate simulation ticks and replaying the previous command in between, so half of all control outputs ignore the newest observations. We present a fast-slow architecture that removes this compromise. A frozen 7B vision-language backbone acts as the slow system, digesting navigation instructions and visual history at low frequency while exposing its per-layer key-value cache as a standing representation of the scene. A lightweight action expert acts as the fast system, attending to this cache and to the current camera frame at every simulation tick to regress waypoints in a single forward pass. Since the cache lags behind the world at deployment, we train the expert under randomized staleness, aligning training with asynchronous execution. On LangAuto-Short routes in CARLA, our system produces fresh control at every 50 ms simulation tick and lifts route completion from 37.0 to 94.0 over the frame-skipping baseline. A frame-skip ablation with the same expert separates the two factors at work: the expert raises the driving score on its own, while per-tick freshness raises completion from 82.1 to 94.0 and cuts red-light violations by a third. Trained on a single town, the expert transfers zero-shot to two unseen towns, holding 84-94% route completion where the baseline reaches 31-41%. It reduces open-loop waypoint error by nearly a factor of four compared to the backbone's own action head, at a per-tick model cost of 32 ms that is independent of history length on a single consumer GPU.