TrafficAlign: Aligning Large Language Models for Traffic Scenario Generation

Recent advances in autonomous driving have emphasized the importance of high-fidelity traffic scenario simulation for safety validation. Early methods relied heavily on handcrafted rules or limited real-world datasets, which restricted diversity and realism. Techniques like Meta-Sim, STRIVE, and SLEDGE introduced probabilistic models and structured representations to generate scenarios, but often lacked regional specificity and scalability. Meanwhile, large language models (LLMs) have shown promise in generating traffic descriptions, yet their outputs often deviate from actual distributions. The challenge remains to synthesize diverse, regionally accurate scenarios that can be systematically validated and used for model training. TrafficAlign addresses this by combining video-based data extraction, formal validation, and model fine-tuning, filling a critical gap in scalable, realistic scenario generation.

The core issue is how to generate traffic scenarios that are both realistic and region-specific, leveraging large-scale video data while ensuring semantic correctness. Existing approaches struggle with limited diversity, poor regional adaptation, or lack of formal validation, leading to scenarios that do not accurately reflect real-world traffic conditions. This hampers the effectiveness of autonomous vehicle testing and model training. Additionally, aligning large language models with regional traffic distributions remains an open problem, as models trained on generic data often produce homogeneous or unrealistic scenarios. Addressing these bottlenecks requires an integrated framework that can extract, validate, and adapt scenarios automatically, ensuring high fidelity and regional relevance.

TrafficAlign's key innovations include: 1) multimodal extraction of traffic scenarios from videos using advanced LLMs, enabling automatic, large-scale data synthesis; 2) formal validation via a domain-specific language (DSL), which ensures semantic completeness and realism; 3) a region-specific fine-tuning process using LoRA, aligning LLMs with local traffic distributions. These innovations collectively enable scalable, high-quality scenario generation that is both diverse and geographically accurate, surpassing prior rule-based or static data-driven methods. The integration of formal validation with adaptive model tuning is particularly novel, providing a robust pipeline for safety-critical applications.

• �� Data collection: Sample traffic videos from YouTube across six diverse regions, extracting frames every 15 seconds to ensure diversity.
• �� Scenario synthesis: Use GPT-4.1 nano with carefully designed prompts to generate detailed natural language descriptions of traffic scenes, including environment, actors, and behaviors.
• �� Formal validation: Convert descriptions into a DSL using GPT-5, checking syntax and semantics for completeness and relevance. Discard invalid scenarios.
• �� Self-refinement: Iteratively improve incomplete scenarios through feedback loops, enhancing data quality.
• �� Model alignment: Fine-tune large models with LoRA on validated scenarios, focusing on regional traffic patterns.
• �� Application: Generate synthetic scenarios for simulation in CARLA, using Scenic scripts derived from descriptions, to evaluate and improve autonomous driving models.

The evaluation involved testing three deep RL autonomous driving models (PPO, SAC, TD3) within the SafeBench platform, using scenarios generated by TrafficAlign and baselines. Metrics included collision rate, route completion, and lane invasion. Experiments covered six regions, assessing scenario challenge level and regional fidelity. Ablation studies compared models with and without LLM alignment, confirming the importance of region-specific fine-tuning. The results demonstrated that TrafficAlign scenarios consistently induced higher challenge levels, leading to better model robustness and safety improvements after fine-tuning. Hyperparameters included 60 training steps with a learning rate of 2e-4 for LoRA tuning.

TrafficAlign scenarios increased collision detection by 10.8% over ChatScene across three models, indicating higher challenge and realism. Fine-tuning with these scenarios reduced collision rates by 36.1%, significantly improving safety metrics. Cross-region evaluations showed high correlation between generated and real traffic distributions, validating regional adaptability. Ablation studies confirmed that without LLM alignment, scenario realism and challenge level decreased, underscoring the importance of the proposed alignment process. Overall, the approach demonstrated superior performance in both scenario challenge and model robustness enhancement.

This framework can be immediately applied to autonomous vehicle safety validation, providing scalable, region-specific testing scenarios that reflect real-world conditions. It reduces reliance on manual scenario design, accelerates testing cycles, and enhances model robustness. Long-term, integrating real-time traffic data and multi-sensor inputs could enable dynamic scenario generation, supporting continuous learning and adaptation of autonomous systems in diverse environments, ultimately leading to safer autonomous driving solutions worldwide.

Dependence on high-quality traffic videos limits applicability in data-scarce regions or adverse weather conditions. DSL validation may not fully capture complex or abnormal behaviors, requiring further enhancement. Fine-tuning large models remains computationally intensive, especially for real-time applications. Future work should focus on improving data efficiency, expanding validation language expressiveness, and reducing training costs to enable broader deployment.