ControlledShifts: Towards Standardizing Robustness Evaluation in Trajectory Prediction Under Distribution Shifts
This paper introduces ControlledShifts, a framework for standardized robustness evaluation in trajectory prediction under distribution shifts, combining a novel bias-controlled data splitting method with a unified performance score.
Key Findings
Methodology
The study proposes ControlledShifts, a systematic framework that employs a shared characterization function (c) to define axes of variation, such as topology or behavior, and a splitting function (f) to control the tail of these axes, creating controlled distribution shifts. This approach re-divides existing datasets—like Waymo Open Motion Dataset—into in-distribution (SEEN) and out-of-distribution (UNSEEN) subsets without additional data collection. The framework encompasses three benchmarks targeting key topological and behavioral shifts: BackgroundAgents, EgoSafeShift, and Environments. To quantify model robustness across these shifts, the authors introduce a unified score that combines prediction quality (performance gain over a naive baseline) and stability (performance preservation relative to a uniform split). Extensive experiments benchmark transformer-based models, revealing capacity-dependent robustness differences and highlighting the importance of multi-dimensional evaluation.
Key Results
- On the Waymo dataset, models exhibit an average increase of over 20% in displacement error (e.g., MinADE rising from 1.953m to 2.625m for Way-Former) under out-of-distribution shifts. Larger models like MTR (27.2M parameters) achieve higher baseline accuracy (score 2.187) but show greater performance degradation (score 0.920), indicating capacity-related sensitivity to distribution shifts.
- The analysis across different axes shows that background agent shifts significantly reduce collision rates but increase displacement errors, revealing a trade-off between safety and accuracy. Safety-related shifts (EgoSafeShift) induce subtle performance drops but cause notable increases in collision rates, exposing latent safety vulnerabilities.
- The unified robustness scores demonstrate that models like Way-Former excel in prediction quality (score ~3.0) but are less stable under unseen shifts, whereas smaller models or naive baselines maintain more consistent performance, emphasizing the importance of multi-faceted robustness evaluation.
Significance
This work addresses a critical gap in autonomous driving research: the lack of standardized, multi-dimensional robustness evaluation under realistic distribution shifts. By formalizing a controllable shift generation process and integrating multiple metrics into a single score, it enables fair comparison of diverse models and fosters development of more reliable trajectory predictors. The framework’s extensibility allows researchers to incorporate new axes of variation, making it a versatile tool for advancing safety-critical applications. Its implications extend beyond autonomous driving, offering a blueprint for robustness assessment in multi-agent systems, robotics, and AI safety domains, ultimately contributing to safer deployment of autonomous systems in unpredictable real-world environments.
Technical Contribution
The paper’s main technical innovation is the ControlledShifts framework, which formalizes the creation of controlled distribution shifts via a shared characterization function (c) and a splitting function (f). This approach allows for flexible, scenario-specific bias induction without additional data collection. Additionally, the authors introduce a unified robustness score that aggregates multiple metrics—such as displacement error, miss rate, and collision rate—across different benchmarks, using a geometric mean to balance prediction quality and stability. The design of three benchmarks (BackgroundAgents, EgoSafeShift, Environments) provides targeted evaluation of models’ robustness to agent behavior, safety-critical scenarios, and topological diversity, respectively. The methodology enables systematic, reproducible comparison across models and shift types, setting a new standard for robustness benchmarking in trajectory prediction.
Novelty
This research is pioneering in its systematic approach to generating controlled, multi-axis distribution shifts directly from existing datasets, avoiding the costly collection of new data. Unlike prior works that focus on single axes or unstructured perturbations, ControlledShifts offers a unified, flexible framework capable of simulating diverse real-world variations. The introduction of a comprehensive robustness score that combines multiple metrics into a single interpretable value is also novel, facilitating cross-model and cross-scenario comparisons. The combination of scenario-specific bias induction, multi-metric evaluation, and extensible benchmark design marks a significant advancement over previous ad hoc or single-metric robustness assessments, establishing a new paradigm in trajectory prediction evaluation.
Limitations
- The framework relies on predefined feature functions (c) and splitting functions (f), which may not capture all relevant axes of real-world distribution shifts, potentially limiting its generality. The choice of features can influence the induced shifts, requiring careful design and validation.
- Computational costs increase with the complexity of the bias induction process and the number of models evaluated, especially for large-scale or high-dimensional scenarios, which may hinder scalability.
- The robustness score primarily measures performance degradation relative to a baseline, but does not explicitly account for interpretability, safety margins, or model explainability, which are critical for deployment in safety-critical systems.
Future Work
Future research should explore more sophisticated, data-driven feature functions to better capture complex real-world shifts. Extending the framework to incorporate additional axes such as weather conditions, sensor noise, or multi-modal behaviors will enhance its applicability. Developing interpretability-aware robustness metrics and integrating safety margins can improve practical deployment. Additionally, applying the framework to other domains like robotics or multi-agent coordination, and exploring adaptive models that can self-adjust to shifts, are promising directions. Finally, creating a standardized benchmark suite for industry adoption could accelerate the development of safer, more reliable autonomous systems.
AI Executive Summary
Autonomous driving relies heavily on accurate trajectory prediction to ensure safety and efficiency. However, existing models, despite在在训练数据丰富的环境中表现优异，面对未知或偏移场景时，性能急剧下降，成为制约其实际应用的主要瓶颈。传统的评估方法多依赖单一指标如误差或碰撞率，缺乏对模型在不同环境变化下的鲁棒性全面衡量，难以指导模型改进。为应对这一挑战，本文提出了ControlledShifts框架，旨在通过系统化的偏移生成机制，实现轨迹预测模型鲁棒性评估的标准化和多维度。
ControlledShifts的核心思想是利用特征函数（c）定义偏移轴，比如道路拓扑、代理行为或安全相关性，然后通过分割函数（f）控制偏移尾部，生成在不同偏移场景下的训练和测试集。该方法无需采集新数据，而是对现有数据集（如Waymo Open Motion Dataset）进行重新划分，形成在分布内（SEEN）和分布外（UNSEEN）两类场景。通过这种方式，可以模拟多种现实中的偏移情况，系统评估模型在不同环境变化中的表现。
论文设计了三套偏移基准：背景代理偏移（BackgroundAgents）、自我安全偏移（EgoSafeShift）和环境拓扑偏移（Environments），分别对应环境中代理行为、决策安全性和道路结构的变化。这些偏移场景涵盖了多样化的环境变化，验证了模型在复杂、未见场景中的鲁棒性。为了量化模型在偏移中的表现，作者提出了统一的鲁棒性得分体系，将预测质量（相对性能提升）和预测稳定性（偏移下性能保持）两个维度融合，形成一个直观、可比的指标。
在广泛的实验中，作者评估了多款Transformer架构（如Way-Former、Autobot、SceneTransformer、MTR）在不同偏移场景下的表现。结果显示，模型容量越大，预测准确性越高，但在偏移环境中的性能波动也越大。具体而言，Way-Former在偏移场景中的平均MinADE从1.953米升至2.625米，性能下降超过34%，而其鲁棒性得分为2.187，显示出较强的预测能力但较差的稳定性。这些发现强调了多轴、多指标评估的重要性，为模型设计提供了明确的优化方向。
该研究的意义在于，为轨迹预测模型的鲁棒性评估提供了一个系统、标准化的工具。通过偏移生成机制和多维指标融合，推动了模型在真实复杂环境中的可靠性提升。其框架具有高度的扩展性，可以加入新的偏移轴和指标，适应不断变化的实际需求。未来，结合强化学习和自适应机制，模型可以在偏移环境中实现自我调整，进一步保障自动驾驶系统的安全性。整体而言，ControlledShifts不仅丰富了学术界对鲁棒性研究的工具箱，也为工业界提供了可操作的评估标准，助力自主系统的安全部署。
Deep Dive
Abstract
Trajectory prediction is central to safety in autonomous driving, yet learning-based predictors tend to degrade sharply when encountering scenarios poorly represented by their training data. Many methods attempt to mitigate distribution shift degradation through data-centric or test-time adaptation approaches; however, they are typically validated along fragmented axes of generalization, leaving the field without a standardized way to compare robustness across shifts a model may encounter. To address this, we introduce ControlledShifts, a framework and benchmark suite that systematically re-splits existing trajectory datasets into in-distribution (seen) and out-of-distribution (unseen) partitions, via a shared characterization-and-splitting formulation, in which a characterization function fixes the axis of variation a benchmark probes and a splitting function fixes how the tail of that axis is withheld. The suite comprises three benchmarks targeting key topological and behavioral distribution shifts. Furthermore, to aggregate multi-dimensional performance metrics across these benchmarks, we propose a unified robustness score that evaluates models along two complementary dimensions: prediction quality (relative performance gain) and prediction stability (performance preservation under shift). We showcase ControlledShifts by benchmarking prominent transformer-based architectures, exposing critical differences in how models of varying capacities handle latent relevance and environmental structure.
References (20)
LongComp: Long-Tail Compositional Zero-Shot Generalization for Robust Trajectory Prediction
Ben Stoler, Jonathan Francis, Jean Oh
SafeShift: Safety-Informed Distribution Shifts for Robust Trajectory Prediction in Autonomous Driving
Ben Stoler, Ingrid Navarro, Meghdeep Jana et al.
Improving the Generalizability of Trajectory Prediction Models with Frenét-Based Domain Normalization
Luyao Ye, Zikang Zhou, Jianping Wang
BITS: Bi-level Imitation for Traffic Simulation
Danfei Xu, Yuxiao Chen, B. Ivanovic et al.
Latent Variable Sequential Set Transformers for Joint Multi-Agent Motion Prediction
Roger Girgis, Florian Golemo, Felipe Codevilla et al.
Another look at measures of forecast accuracy
Rob J Hyndman, A. Koehler
KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradients
Niklas Hanselmann, Katrin Renz, Kashyap Chitta et al.
nuScenes: A Multimodal Dataset for Autonomous Driving
Holger Caesar, Varun Bankiti, Alex H. Lang et al.
Scene Transformer: A unified architecture for predicting future trajectories of multiple agents
Jiquan Ngiam, Vijay Vasudevan, Benjamin Caine et al.
Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?
Angelos Filos, P. Tigas, R. McAllister et al.
Core Challenges in Embodied Vision-Language Planning
Jonathan M Francis, Nariaki Kitamura, Felix Labelle et al.
Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset
S. Ettinger, Shuyang Cheng, Benjamin Caine et al.
On Adversarial Robustness of Trajectory Prediction for Autonomous Vehicles
Qingzhao Zhang, Shengtuo Hu, Jiachen Sun et al.
LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization
Xueyang Zhou, Yangming Xu, Guiyao Tie et al.
Motion Transformer with Global Intention Localization and Local Movement Refinement
Shaoshuai Shi, Li Jiang, Dengxin Dai et al.
SEAL: Towards Safe Autonomous Driving via Skill-Enabled Adversary Learning for Closed-Loop Scenario Generation
Ben Stoler, Ingrid Navarro, Jonathan Francis et al.
What Went Wrong? Closing the Sim-to-Real Gap via Differentiable Causal Discovery
Peide Huang, Xilun Zhang, Ziang Cao et al.
ScenarioCharacterization: A Modular Toolkit for Characterizing Safety across Trajectory Datasets
Ingrid Navarro, Yutong Duan, Jonathan Francis et al.
RCG: Safety-Critical Scenario Generation for Robust Autonomous Driving via Real-World Crash Grounding
Ben Stoler, Julia S. Yang, Jonathan Francis et al.
CausalAgents: A Robustness Benchmark for Motion Forecasting
R. Roelofs, Liting Sun, Benjamin Caine et al.