Long-term Traffic Simulation via Structured Autoregressive Modeling
RosettaSim employs structured autoregressive modeling with attention mechanisms and semantic retrieval, achieving state-of-the-art long-term traffic simulation with a correlation of r=0.83.
Key Findings
Methodology
This work leverages large-scale sequence models, such as pretrained Transformers, to encode scene topology, agent states, and spawning intents into variable-length autoregressive token sequences. By exploiting the Zipf distribution of traffic motion tokens, the model treats traffic dynamics as a structured language, enabling long-range dependency modeling. The framework integrates a scene encoder, motion token generator, and autoregressive agent spawn module, trained end-to-end with cross-entropy loss. A semantic retrieval-based evaluation (RTE) retrieves similar real-world scenarios from a pretrained VAE latent space, ensuring robust long-term assessment. The approach supports dynamic agent entry/exit, multi-agent interactions, and scene consistency.
Key Results
- On the Waymo Open Motion Dataset (WOMD), RosettaSim outperforms existing methods in short-term (8s) and long-term simulation, with a 2-3% improvement on WOSAC metrics and a long-horizon correlation coefficient of 0.83, surpassing previous approaches (0.74). Ablation studies show that frozen pretrained models still provide strong structural priors, and the semantic retrieval enhances evaluation reliability.
- The model demonstrates high adaptability across different LLM architectures, notably Qwen2.5B, with minimal fine-tuning, indicating robustness of the pretraining prior. The retrieval-based evaluation framework significantly improves the correlation with human judgment and real-world scenarios.
- Extensive experiments validate the model's ability to generate realistic multi-agent interactions over extended horizons, maintaining scene fidelity and interaction consistency, thus addressing key challenges in long-term traffic simulation.
Significance
This research addresses fundamental challenges in long-term traffic simulation: modeling sustained multi-agent interactions with dynamic populations. By integrating large-scale sequence models with semantic retrieval, it advances the fidelity, stability, and evaluability of traffic simulations. This breakthrough supports safer autonomous driving testing, urban planning, and intelligent transportation systems. The approach bridges NLP-inspired modeling with multi-agent dynamics, opening new avenues for AI-driven traffic management and simulation, crucial for future smart cities.
Technical Contribution
The paper introduces RosettaSim, a unified framework combining structured autoregressive sequence modeling with attention mechanisms and semantic retrieval. It exploits Zipf’s law in traffic token distributions, leveraging pretrained Transformer models as structural priors. The architecture supports variable agent populations, dynamic scene evolution, and long-range dependencies. Additionally, the RTE evaluation framework employs a pretrained VAE to retrieve semantically similar scenarios, providing a context-aware, robust metric for long-horizon simulation fidelity. These innovations significantly enhance the modeling capacity and evaluation reliability of traffic simulators.
Novelty
This work is the first to conceptualize traffic dynamics as a structured language, utilizing Zipf distribution properties of traffic tokens and pretrained large models to address long-horizon simulation challenges. The integration of semantic retrieval for evaluation is novel, providing a context-aware, statistically grounded assessment method. The combination of these techniques results in a model that surpasses existing methods in both short- and long-term fidelity, representing a significant step forward in traffic simulation research.
Limitations
- The model’s performance diminishes in highly unpredictable or rare scenarios, such as accidents or unusual weather conditions, due to limited training data diversity.
- High computational costs associated with large pretrained models may hinder real-time deployment in resource-constrained environments.
- Multi-modal data integration (e.g., sensor inputs) remains an open challenge, requiring further research to enhance scene understanding and robustness.
Future Work
Future directions include integrating multi-modal sensor data for richer scene understanding, applying reinforcement learning to optimize traffic policies, and extending模型到多城市、多场景的泛化能力。此外，研究者还计划探索模型在突发事件中的应对策略，以及提升模型的实时性和可扩展性，以实现更广泛的自动驾驶应用。
AI Executive Summary
Traffic simulation is a cornerstone of autonomous driving development, yet existing methods struggle with long-term, multi-agent scenarios involving dynamic entry and exit of vehicles. Traditional models excel in short-term predictions but falter over extended horizons, often due to the difficulty in balancing scene generation and motion consistency. This paper introduces RosettaSim, a novel framework that leverages large-scale pretrained Transformer models to encode traffic scene topology, vehicle states, and spawning intents into structured token sequences. Inspired by linguistic principles, the model exploits the Zipf distribution characteristic of traffic motion tokens, viewing traffic as a structured language. This analogy enables the model to inherit the long-range dependency modeling capabilities of NLP transformers, effectively capturing complex multi-agent interactions over extended periods. The architecture combines scene encoders, attention mechanisms, and autoregressive modules to support dynamic agent populations, ensuring scene fidelity and interaction realism. To evaluate long-term simulation fidelity, the authors propose RTE, a retrieval-based metric that uses a pretrained VAE to find semantically similar real-world scenarios, providing a context-aware benchmark. Experimental results on the Waymo Open Motion Dataset demonstrate that RosettaSim surpasses existing approaches in both short- and long-term metrics, with a correlation coefficient of 0.83, significantly higher than previous methods. The combination of structural priors, semantic retrieval, and autoregressive modeling marks a substantial advancement in traffic simulation, promising safer and more reliable autonomous driving systems. Despite its strengths, the approach faces challenges in rare event modeling and computational efficiency, guiding future research directions towards multi-modal integration and real-time deployment.
Deep Dive
Abstract
Interactive traffic simulation is a vital world model for autonomous driving. A central challenge in long-horizon simulation is modeling sustained multi-agent interactions, which is further exacerbated by dynamic token cardinality as agents continuously enter and exit the scene. In this work, we propose that the solution lies in the synergy between the architectural inductive biases and statistical priors of large-scale sequence models, e.g., Large Language Models (LLMs). Our probing experiments reveal that the transferability of attention mechanisms and the distributional consistency between motion tokens and natural language enable small-scale, heavily frozen LLMs to rapidly adapt to traffic modeling. Building on this insight, we introduce RosettaSim, a unified framework that projects scene topology, agent states, and spawning intents into a structured autoregressive stream with variable length, achieving both strong short-term accuracy and stable long-horizon simulation fidelity. Furthermore, evaluating extended rollouts presents yet another hurdle, as one-to-one agent correspondence inevitably fades over time. To address this, we introduce Retrieval-based Traffic Evaluation (RTE), which retrieves semantically similar real-world scenarios as context-aware reference anchors. Experiments on the Waymo Open Sim Agent Challenge (WOSAC) demonstrate that RosettaSim achieves state-of-the-art performance in both short- and long-term simulation. Furthermore, RTE exhibits a stronger correlation with standard metrics ($r=0.83$) than existing approaches ($r=0.74$), indicating improved alignment with long-horizon simulation fidelity.