Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset

Recent advances in autonomous driving have emphasized the importance of accurate motion prediction for safe navigation. Early datasets like KITTI and Waymo Open Dataset provided static scene annotations but lacked detailed interaction scenarios. More recent datasets such as NuScenes and Argoverse introduced richer object taxonomy but still focused on single-agent trajectories. The challenge remains in modeling complex multi-agent interactions, especially in urban environments with merges, crossings, and unprotected turns. Existing models often treat agents independently, limiting their ability to capture interdependencies, which are critical for safe planning. The need for large-scale, high-quality, interaction-rich datasets is evident to push the frontier of joint prediction models.

The core challenge lies in accurately predicting the future trajectories of multiple agents engaged in complex interactions within urban traffic scenes. Traditional datasets and models fall short in capturing the nuanced behaviors arising from inter-agent influences, leading to suboptimal planning and safety risks. Moreover, existing benchmarks lack comprehensive metrics to evaluate joint prediction performance, making it difficult to compare models effectively. The problem is compounded by the scarcity of large, diverse datasets that include detailed annotations of interaction scenarios involving vehicles, pedestrians, and cyclists across varied road geometries. Addressing this gap is essential for advancing interaction-aware motion forecasting.

This work introduces several key innovations: 1) a large-scale, diverse dataset capturing multi-agent interactions with detailed annotations, enabling training of joint prediction models; 2) a high-accuracy offboard perception pipeline that generates high-quality 3D bounding boxes and maps, reducing perception noise; 3) novel evaluation metrics tailored for multi-agent scenarios, including joint ADE, overlap rate, and mAP, providing comprehensive performance assessment; 4) baseline models that incorporate rich scene context such as HD maps, traffic signals, and high-order interactions, demonstrating the benefits of scene-aware modeling. These innovations collectively address limitations of prior datasets and evaluation methods, fostering more realistic and robust motion prediction research.

• �� Data collection: Using autonomous vehicles across six US cities, scenes are mined for interactions like merges, turns, and crossings based on semantic predicates. • Automatic labeling: An offboard detection and tracking system employs 3D LiDAR data to generate high-precision bounding boxes, followed by a refinement network ensuring temporal consistency. • Scene segmentation: Each 20-second scene is split into 9.1-second segments, capturing complete interaction events. • Annotation: Interactions are explicitly labeled, with semantic predicates guiding the selection of scenarios. • Model inputs: Encoded scene features include vehicle, pedestrian, cyclist states, 3D map polylines, traffic signals, and high-order interaction graphs. • Prediction: K=6 trajectories per agent are generated using MLP decoders, trained with min-of-k loss. • Evaluation: Metrics like minADE, miss rate, overlap rate, and mAP are computed at 8s horizon, with ablation studies on feature importance.

The dataset is split into training (70%), validation (15%), and test (15%) sets, ensuring diverse interaction scenarios. Baseline models include constant velocity, LSTM encoders, and scene-aware deep models. Experiments evaluate the impact of map, traffic signals, and interaction features on prediction accuracy. Metrics such as minADE, miss rate, and mAP are used to compare models across different scene complexities. Ablation studies analyze the contribution of each feature type. Results demonstrate that incorporating scene context and high-order interactions significantly improves long-term prediction performance, especially in complex scenarios like merges and crossings.

Models leveraging rich scene features outperform simple baselines, with a 20% reduction in minADE and 15% increase in mAP over 8 seconds. Joint prediction models show superior performance in multi-agent scenarios, with a notable decrease in overlap rate and miss rate. The dataset's diversity enables models to generalize across various behaviors, with the best models achieving a joint minADE of approximately 1.36m for vehicles and 1.54m for pedestrians. These results validate the dataset's effectiveness in fostering interaction-aware motion prediction research.

The dataset and models support advanced autonomous driving functions such as path planning, collision avoidance, and interaction modeling. They enable safer navigation in complex urban environments by accurately predicting multiple agents' future behaviors. Long-term prediction capabilities can improve decision-making in scenarios like merging, crossing, and unprotected turns, ultimately enhancing passenger safety and traffic efficiency. The framework also facilitates the development of real-time perception and prediction systems, bridging research and industry needs.

While the dataset covers diverse urban scenarios, it is primarily based on US cities, limiting regional applicability. The models still face challenges in rare or highly dynamic interactions, and high computational costs of training large models may hinder deployment. The reliance on offline perception for annotations, though high quality, may not fully replicate real-time onboard perception noise. Future work should focus on expanding geographic diversity, improving model efficiency, and integrating online perception for real-world deployment.