ForgeDrive: Bidirectional Cross-Conditioning for Unified Visual-Action Generation in Autonomous Driving
ForgeDrive achieves 90.3 EPDMS on NAVSIM by unifying driving simulation and planning via visual-action cross-conditioning.
Key Findings
Methodology
ForgeDrive employs an autoregressive diffusion framework with visual-action cross-conditioning, leveraging UniDiffuser-style noise scheduling to iteratively generate future frames and actions while predicting ego status.
Key Results
- Achieved 90.3 EPDMS on NAVSIM v2, outperforming DreamAD and Latent-WAM with superior planning performance.
- Video generation quality leads with 69.2 FVD, supporting 120-second rollouts, far exceeding competitors.
- Trajectory prediction reaches 0.11m ADE, comparable to DPVO (0.09m), a dedicated visual odometry method.
Significance
This research unifies driving simulation, planning, and visual odometry, addressing cascading errors from visual generation to action planning, offering a robust solution for autonomous driving.
Technical Contribution
ForgeDrive introduces decoupled diffusion timesteps and bidirectional cross-conditioning, enabling deep coupling between visual and action modalities. The act-then-imagine strategy significantly improves generation quality.
Novelty
ForgeDrive pioneers the act-then-imagine paradigm, reducing visual error propagation into action planning compared to traditional imagine-then-act approaches.
Limitations
- Struggles with high-dynamic scenarios, leading to trajectory drift.
- Requires significant computational resources, resulting in longer inference times.
Future Work
Future research could explore more efficient diffusion architectures and robust prediction in complex dynamic environments.
AI Executive Summary
Existing autonomous driving methods often follow an imagine-then-act paradigm, where errors in visual generation cascade into action planning. ForgeDrive introduces a unified autoregressive diffusion framework that employs visual-action cross-conditioning and adopts an act-then-imagine strategy to iteratively generate future frames and actions while predicting ego status.
In NAVSIM benchmarks, ForgeDrive achieves 90.3 EPDMS, outperforming strong baselines such as DreamAD and Latent-WAM. Additionally, it leads in video generation quality with 69.2 FVD and supports long-horizon predictions up to 120 seconds, showcasing exceptional robustness and accuracy.
Despite its strong performance, ForgeDrive faces challenges in high-dynamic scenarios and demands significant computational resources. Future work could focus on improving efficiency and adaptability to complex environments.
Deep Analysis
Background
Autonomous driving has advanced significantly, particularly with world-model-based methods that predict future scenes to enhance planning. However, existing methods suffer from cascading errors due to the imagine-then-act paradigm, impacting trajectory accuracy.
Core Problem
Cascading errors from visual generation to action planning remain a critical bottleneck, especially for long-horizon predictions where inaccuracies accumulate over time.
Innovation
ForgeDrive introduces an act-then-imagine paradigm, decouples diffusion timesteps for visual and action modalities, and employs UniDiffuser noise scheduling to achieve deep coupling and mutual refinement between modalities.
Methodology
- �� Utilizes an autoregressive diffusion framework to iteratively generate future frames and actions.
- �� Implements UniDiffuser noise scheduling for visual-action cross-conditioning.
- �� Predicts ego status at each timestep to ensure physical consistency.
- �� Updates historical context via a sliding window for long-horizon generation.
Experiments
Experiments were conducted on the NAVSIM benchmark using PDMS and EPDMS metrics, alongside FVD for video quality evaluation, comparing against multiple baselines.
Results
ForgeDrive achieves 90.3 EPDMS on NAVSIM v2, surpassing DreamAD and Latent-WAM; video generation quality leads with 69.2 FVD; trajectory prediction reaches 0.11m ADE, comparable to DPVO.
Applications
Direct applications include autonomous driving systems for trajectory planning and long-horizon predictions, such as urban traffic management.
Limitations & Outlook
Challenges include high-dynamic scenario prediction and computational demands, leading to longer inference times.
Plain Language Accessible to non-experts
Imagine ForgeDrive as a smart assistant in a kitchen. It first predicts your next cooking action, like chopping vegetables, and then adjusts the recipe steps based on that action. This act-then-imagine approach avoids mistakes caused by incorrect ingredient assumptions.
ELI14 Explained like you're 14
Think of playing a racing game. ForgeDrive is like a super AI teammate that predicts your next turn first, then generates the upcoming track based on that move. It ensures you always stay ahead in the race!
Glossary
Diffusion Model
A generative model that creates data by iteratively denoising. Used for generating future frames and actions.
Central to ForgeDrive's autoregressive framework.
Autoregressive
A method of generating data step-by-step, where each step depends on the previous output.
Used for iterative frame-action generation.
UniDiffuser
A multi-modal diffusion framework enabling cross-modal conditional generation.
Facilitates visual-action cross-conditioning in ForgeDrive.
EPDMS
A metric for evaluating autonomous driving planning performance.
Used in NAVSIM benchmark evaluation.
FVD
A metric for assessing video generation quality, measuring realism and temporal coherence.
Used to compare video generation fidelity.
Open Questions Unanswered questions from this research
- 1 How to improve robustness in high-dynamic scenarios?
- 2 Can computational efficiency be enhanced for faster inference?
Applications
Immediate Applications
Autonomous Driving Planning
Enhances trajectory accuracy and robustness through deep visual-action coupling.
Urban Traffic Management
Supports long-horizon predictions for optimizing traffic flow.
Long-term Vision
Fully Autonomous Systems
Enables end-to-end driving simulation and planning, advancing fully autonomous driving technology.
Abstract
World-model-based autonomous driving endows the model with the ability to understand scene evolution. Yet this promise is undermined by the prevailing imagine-then-act paradigm, which allows errors from the more challenging visual generation stage to cascade into action planning. We introduce ForgeDrive, a unified autoregressive diffusion framework with visual-action cross-conditioning that closes this gap through act-then-imagine paradigm. ForgeDrive factorizes the future as a sequence of per-timestep frame-action pairs, intertwining each action with its corresponding visual observation. During training, we decouple the diffusion timesteps of the two modalities and introduce a UniDiffuser-style noise scheduler to get the ability to infer either modality from its counterpart and deepen understanding of relationships between images and actions. At inference, we propose a novel act-then-imagine inference paradigm, and find that at each step, action generation is a capability internalized during training, requiring no clean future frame as a prerequisite at inference time; instead, the generated action can improve the accuracy of future frame generation, which in turn enhances the quality of the next action. Additionally, we augment each step with future ego-status prediction, further sharpening planning ability. Extensive experiments on NAVSIM demonstrate that ForgeDrive not only unifies driving simulation, planning, and visual odometry into a single model, but also outperforms existing strong planners without any post-training strategy.