From Attacks to Curricula: Learnability-Guided Adversarial Training for Safe Autonomous Driving

Autonomous driving has made major progress in end-to-end policy learning, but long-tail safety-critical events remain the central obstacle. Real-world crashes and near-misses are extremely rare, so naturalistic data alone cannot cover the corner cases that matter most. This has motivated simulation-based adversarial scenario generation and closed-loop adversarial training. Existing generators include optimization-based trajectory attacks, reinforcement-learning-based search, and deep generative models such as diffusion-based methods. These approaches are useful for exposing vulnerabilities, but their common objective is collision maximization. That can produce physically plausible yet strategically hopeless scenes, where the ego vehicle has no realistic chance to avoid failure. In parallel, scenario-based curriculum learning has evolved from manual staging to automated sampling, but most schedulers remain based on static heuristics, past rewards, or temporal-difference signals. They lag behind the rapidly changing ability of the driving policy.

The paper tackles two coupled bottlenecks. First, how can we generate adversarial traffic scenes that are challenging enough to matter but still resolvable for some policy? Unsolvable scenarios waste training capacity because they provide little action-discriminative feedback. Second, how can we estimate the current policy’s capability on a given scenario before running expensive closed-loop simulation? This is difficult because the policy updates continuously, its competence is non-linear across scenarios, and the scenario space is high-dimensional and dynamic. If sampling does not match current capability, easy scenarios become overrepresented and learning slows down. The result is sample inefficiency, delayed convergence, and often conservative behavior.

AlignADV contributes three main innovations. First, it recasts adversarial generation as preference alignment: rather than directly solving a hard constrained min-max problem, it uses DPO to fine-tune a pretrained generator toward resolvable yet safety-critical outputs. This makes the method modular and easy to plug into existing generators. Second, it introduces the behavioral fingerprint, an algorithm-agnostic representation of the evolving ego policy. By probing the policy on selected states and summarizing its response pattern, the method captures competence in a compact form. Third, it couples the generator and predictor into capability-aligned curriculum sampling. The predictor estimates success probability for candidate scenarios, and the sampler prioritizes scenarios near the policy’s vulnerability boundary. This is a forward-looking alternative to replaying historical difficulty statistics.

• �� Problem setup: Safe driving is formulated as an MDP with state space, action space, transition dynamics, reward, and discount factor. The policy π_θ maximizes the expected discounted return J(π_θ,ϕ) over a scenario library Φ.

• �� Adversarial generation: A generator G_ψ perturbs a base scenario ϕ into an adversarial variant ϕ'. The usual objective is to minimize the policy’s return, but AlignADV adds a solvability constraint max_{π∈Π}J(π,ϕ')≥δ_safe to ensure at least one policy can handle it.

• �� Preference alignment with DPO: The method constructs preference pairs between candidate scenarios, guided by expert or rule-based judgments of resolvability. DPO then post-trains the generator so that its output distribution shifts toward preferred, solvable adversarial trajectories.

• �� Behavioral fingerprint: Instead of tracking only past reward, the framework extracts fingerprint features from the driving policy’s behavior on probe states. These features act as an algorithm-agnostic signature of the current competence state.

• �� Multi-modal capability prediction: A global encoder models the fingerprint, a local context encoder models the scenario, and cross-attention fuses them to predict success probability C_ω(π_θ,\tilde{ϕ}) before simulation.

• �� Dynamic curriculum sampling: The predicted success probabilities define P_{θ,ω}(\tilde{ϕ}), which upweights scenarios that are solvable but currently difficult. The training loop then repeats online as the policy evolves, creating a closed feedback cycle of generation, prediction, sampling, and optimization.

Experiments are conducted on the Waymo Open Motion Dataset, with evaluation centered on closed-loop policy training. The paper compares AlignADV with baseline adversarial-training and curriculum-sampling approaches, measuring convergence speed, collision rate, and route completion rate. It also evaluates performance under both naturalistic and adversarial traffic conditions. In addition, ablation studies isolate the effect of DPO-based resolvability alignment, the behavioral fingerprint, and capability-aligned sampling to verify which component drives the gains. The experimental design emphasizes online closed-loop learning rather than one-shot offline scenario generation, which makes the efficiency claims especially meaningful.

The headline result is a reduction of up to 40.6% in training steps relative to baselines, showing that the method improves not only safety but also optimization efficiency. The paper also reports lower collision rate and higher route completion rate in both normal and adversarial traffic, indicating that AlignADV does not merely overfit to attack scenarios. Instead, it improves the policy’s real driving competence. The ablation studies are equally important: removing DPO makes generated scenarios less reliably solvable; removing the capability predictor or curriculum alignment weakens sample efficiency and final performance. Together, the results support the paper’s central claim that resolvability and capability alignment are both necessary.

In practice, AlignADV could be integrated into autonomous-driving simulation stacks as a scenario triage and scheduling layer. A generator produces challenging candidate scenes, while the predictor decides which ones should be sampled next for policy updates. This would reduce wasted rollouts and improve version-to-version iteration speed. The same idea could also improve regression testing, where engineers want scenes that are both hard and informative rather than merely catastrophic. For industry, the main value is a more efficient safety-improvement loop that keeps pace with rapidly changing driving stacks.

The framework still depends on high-quality preference data and a reliable capability predictor, so errors in either stage can degrade the curriculum. If the policy distribution shifts rapidly, the behavioral fingerprint may become stale. The paper’s evidence is also primarily on Waymo, so cross-dataset and cross-domain robustness remain open. Future improvements may come from better online calibration, richer multi-agent fingerprints, and broader validation across weather, map, and traffic style shifts.