ASSCG: Just-Right Gating over Chattering for Fast-Slow LLM Planning in Autonomous Driving
ASSCG uses RWKV for adaptive gating in fast-slow systems, achieving +2.28 score and 60% latency reduction on nuPlan Hard20.
Key Findings
Methodology
ASSCG is an architecture-agnostic control gate leveraging an RWKV backbone for frame-level Query/Cache/Drop decisions. It is trained via supervised fine-tuning and compute-aware GRPO reinforcement learning.
Key Results
- On nuPlan Hard20, integrating ASSCG into AsyncDriver improved the score to 67.28 (+2.28) while reducing average end-to-end latency by ~60%.
- On NAVSIM, ASSCG in a RecogDrive-based system achieved 91.4 PDMS (+0.6) and increased average speed by ~25%.
- Ablation studies confirmed the effectiveness of RWKV for long-horizon gating and the importance of the Query/Cache/Drop framework.
Significance
This work addresses the inefficiencies of manually designed triggering rules in fast-slow planners by introducing ASSCG, which dynamically optimizes slow-system invocation. It has significant implications for improving autonomous driving planning under resource constraints.
Technical Contribution
The paper introduces a novel RWKV-based gating mechanism for long-horizon temporal modeling in partially observable environments. It formalizes slow-system invocation as a sequential decision problem and optimizes it with compute-aware reinforcement learning, achieving a superior balance of efficiency and performance.
Novelty
ASSCG is the first to use RWKV for gating in fast-slow systems, introducing a Query/Cache/Drop framework that outperforms fixed or heuristic triggering strategies in both efficiency and accuracy.
Limitations
- The method relies on high-quality supervised labels, which may limit generalizability to new domains.
- Performance in highly complex scenarios may be constrained by the underlying fast and slow systems.
- The effectiveness of RWKV may depend on its parameter size and the diversity of training data.
Future Work
Future work could explore ASSCG's adaptability to other fast-slow architectures, robustness in extreme scenarios, and integration with multimodal inputs for enhanced decision-making.
AI Executive Summary
Autonomous driving planners must make real-time decisions in complex, dynamic environments. While large language models (LLMs) offer advanced reasoning capabilities, their high computational cost makes per-frame invocation impractical. Existing fast-slow planners rely on fixed or heuristic triggering rules, which often lead to inefficiencies or suboptimal performance.
To address this, the authors propose the Adaptive Slow-System Control Gate (ASSCG), a frame-level gating mechanism powered by an RWKV backbone. ASSCG dynamically decides whether to Query the slow system, Cache existing guidance, or Drop outdated guidance. It is trained using supervised fine-tuning followed by compute-aware GRPO reinforcement learning. The method was tested on two fast-slow architectures: AsyncDriver on nuPlan Hard20, where it improved the score to 67.28 (+2.28) with a 60% latency reduction, and a RecogDrive-based system on NAVSIM, achieving 91.4 PDMS (+0.6) and a 25% speed increase.
This research demonstrates the potential of ASSCG to optimize fast-slow collaboration in resource-constrained scenarios, offering a novel approach to balancing performance and efficiency. However, its reliance on high-quality training data and challenges in extreme scenarios highlight areas for future improvement and exploration.
Deep Analysis
Background
Recent advancements in autonomous driving planning include learning-based methods and standardized benchmarks. However, current planners struggle with complex, dynamic, long-tail scenarios. In parallel, LLMs have shown strong reasoning and generalization capabilities, motivating their integration into planning systems for high-level guidance.
Core Problem
LLMs are computationally expensive, making per-frame invocation infeasible for real-time planning. Existing fast-slow systems rely on fixed or heuristic triggering rules, which often fail to adapt to dynamic scene complexity, leading to inefficiencies or degraded performance.
Innovation
The paper introduces ASSCG, an adaptive gating mechanism for fast-slow systems. Key innovations include:
- �� A Query/Cache/Drop framework for precise slow-system scheduling.
- �� Use of RWKV for long-horizon temporal modeling and low-latency decisions.
- �� Compute-aware GRPO reinforcement learning to optimize task-level driving metrics under resource constraints.
Methodology
Key steps in ASSCG include:
- �� Extracting scene features using a Vector Map Encoder and modeling temporal context with an RWKV backbone.
- �� Generating frame-level actions (Query, Cache, Drop) to manage slow-system interactions.
- �� Training the gate via supervised fine-tuning on pseudo labels, followed by compute-aware GRPO reinforcement learning.
- �� Integrating ASSCG into fast-slow systems, where cached slow guidance is fused into the fast planner via cross-attention.
Experiments
Experiments were conducted on nuPlan Hard20 and NAVSIM datasets, evaluating ASSCG in AsyncDriver and a RecogDrive-based system. Fixed query schedules served as baselines, and ablation studies assessed the impact of RWKV and the Query/Cache/Drop framework. Metrics included planning scores, latency, and speed.
Results
On nuPlan Hard20, ASSCG improved AsyncDriver's score to 67.28 (+2.28) while reducing latency by 60%. On NAVSIM, it achieved 91.4 PDMS (+0.6) and increased speed by 25%. Ablations confirmed RWKV's effectiveness for long-horizon gating and the importance of the Query/Cache/Drop mechanism.
Applications
ASSCG is suitable for resource-constrained autonomous driving scenarios, enhancing planning efficiency and performance. Its flexible gating mechanism can also be extended to other domains requiring fast-slow collaboration, such as robotics and industrial automation.
Limitations & Outlook
ASSCG's performance depends on high-quality training data and labels, which may limit its generalizability. Its robustness in highly complex scenarios and its reliance on the underlying fast and slow systems require further investigation.
Plain Language Accessible to non-experts
Imagine you're managing a busy kitchen. You have a fast sous-chef who can chop vegetables quickly but sometimes makes mistakes. You also have a master chef who gives excellent advice but takes time to think. You need to decide when to ask the master chef for help and when to let the sous-chef continue.
ASSCG acts like a smart kitchen manager. It decides when to consult the master chef (Query), when to reuse the last advice (Cache), or when to ignore the advice because it might be wrong (Drop). For example, if you're chopping carrots and the master chef's advice is still valid, you can Cache it. If you're unsure about a tricky recipe, you Query the master chef. If the advice is outdated, you Drop it.
This way, the kitchen runs efficiently, balancing speed and quality. Similarly, ASSCG helps autonomous vehicles make smart decisions, ensuring safety and efficiency on the road.
ELI14 Explained like you're 14
Imagine you're playing a racing game. You have two helpers: one is super fast but makes mistakes, and the other is super smart but takes time to think. You can't always use the slow helper because it would slow you down, but using only the fast one might make you crash!
So, you need a system to decide when to use which helper. That's what ASSCG does! It looks at the situation and decides: should it ask the slow helper for advice (Query), keep using the last advice (Cache), or ignore the advice because it's outdated (Drop)?
For example, if you're driving on a straight road, the fast helper is fine. But if you're approaching a tricky turn, ASSCG might ask the slow helper for guidance. If the slow helper gives bad advice, ASSCG can ignore it and stick with the fast helper.
This way, you can drive fast and safely without wasting time or crashing. Cool, right?
Glossary
ASSCG (Adaptive Slow-System Control Gate)
A dynamic gating mechanism for frame-level decisions on slow-system invocation.
Used to optimize resource usage and performance in fast-slow systems.
RWKV (Recurrent Weighted Key-Value)
A lightweight recurrent network for long-horizon temporal modeling.
Serves as the backbone for ASSCG's gating decisions.
Query
Action to invoke the slow system and refresh the buffer.
Used when new high-level guidance is needed.
Cache
Action to reuse the buffer without querying the slow system.
Avoids redundant computation during Equivalent Intervals.
Drop
Action to ignore the buffer and use a null feature.
Prevents stale guidance from affecting planning during Failure Intervals.
Open Questions Unanswered questions from this research
- 1 How can ASSCG's robustness in extreme scenarios be improved?
- 2 How can reliance on high-quality supervised labels be reduced to enhance generalizability?
- 3 Can ASSCG be extended to other multimodal fast-slow systems?
Applications
Immediate Applications
Autonomous Driving Optimization
Enhances planning efficiency and performance in resource-constrained autonomous driving scenarios.
Robotic Path Planning
Applies to robotics for efficient path planning and dynamic adjustments.
Long-term Vision
General Intelligent Collaboration
Extends to multimodal AI systems for broader intelligent collaboration.
Abstract
Large language models (LLMs) can improve autonomous driving planning but are costly to query online, and existing fast-slow planners often rely on hand-designed triggering rules that either over-call the slow system or call it at the wrong times. We formulate slow-system invocation as a resource-aware sequential decision problem and propose the Adaptive Slow-System Control Gate (ASSCG), which makes frame-level Query/Cache/Drop decisions to refresh, reuse, or suppress slow guidance. ASSCG uses an RWKV backbone for efficient long-horizon gating and is trained with supervised fine-tuning followed by GRPO-style compute-aware reinforcement fine-tuning. We apply ASSCG to two different fast-slow architectures: (i) AsyncDriver on nuPlan Hard20 closed-loop evaluation, where ASSCG improves score to 67.28 (+2.28) while reducing average end-to-end inference latency by 60%; and (ii) a RecogDrive-based dual system that we build by replacing its original VLM-2B module with a lightweight ViT-based fast planner and adding an LLM slow planner, evaluated on NAVSIM, where ASSCG achieves 91.4 PDMS (+0.6) and increases average speed by 25%. The project page, including video visualizations and additional results, is available at https://williamxuanyu.github.io/asscg/.