LongComp: Long-Tail Compositional Zero-Shot Generalization for Robust Trajectory Prediction
Proposes LongComp with scene factorization to enhance zero-shot generalization in autonomous driving trajectory prediction, reducing OOD gap to 2.8%.
Key Findings
Methodology
This paper introduces a safety-informed scene factorization framework that decomposes driving scenarios into discrete ego and social axes. Drawing on compositional zero-shot learning (CZSL) principles from computer vision, it constructs out-of-distribution (OOD) test sets by holding out novel combinations of these axes. Scene features are extracted and encoded via autoencoders, then clustered with k-means to generate scene categories. To improve generalization, task-modular gating networks (TMN) and an auxiliary difficulty prediction head are integrated into trajectory prediction models. Experiments on the Waymo Open Motion Dataset (WOMD) demonstrate that baseline models exhibit performance gaps of 5.0% (closed-world) and 14.7% (open-world), which are significantly reduced to 2.8% and 11.5% respectively after applying the proposed methods, while maintaining in-distribution accuracy.
Key Results
- On WOMD, the baseline model's OOD performance gap decreased from 5.0% to 2.8% in closed-world scenarios and from 14.7% to 11.5% in open-world scenarios, showing substantial robustness improvement. The combined use of scene factorization, TMN, and difficulty prediction led to an overall performance boost of 4.0% and 1.2% in in-distribution settings. Ablation studies confirmed that the synergy of these strategies was essential for optimal results.
- Scene factorization enabled the model to better understand behavior variations across different traffic contexts, especially unseen combinations, demonstrating strong zero-shot generalization. The models performed consistently across diverse behavior types and complex scenarios, validating the approach's effectiveness.
- Ablation experiments indicated that the task-modular gating network and difficulty prediction head jointly contributed most to performance gains, with each component providing complementary benefits. The results highlight the importance of multi-strategy integration for robust out-of-distribution prediction.
Significance
This work addresses a critical bottleneck in autonomous driving: the inability of models to handle rare, safety-critical scenarios. By introducing a scene factorization framework and zero-shot evaluation, it offers a pathway to develop models that are more robust and reliable in real-world, unpredictable traffic environments. The approach not only enhances safety but also advances the theoretical understanding of generalization in complex behavior prediction tasks. Its implications extend beyond autonomous driving, providing insights into handling long-tail distributions in other sequential decision-making systems.
Technical Contribution
The paper's core innovation lies in adapting CZSL concepts to trajectory prediction, employing safety-informed scene factorization to create meaningful, disentangled axes. It develops an autoencoder-based discretization and clustering pipeline to generate long-tail test scenarios. The integration of task-modular gating networks and a difficulty-aware auxiliary head into baseline models represents a novel approach to improving out-of-distribution robustness. These strategies jointly enable models to better generalize to unseen behavior combinations, setting new benchmarks for safety-critical applications.
Novelty
This is the first application of compositional zero-shot learning principles to autonomous driving trajectory prediction, utilizing safety-informed scene factorization to generate challenging long-tail evaluation settings. Unlike prior distribution shift methods, it explicitly disentangles ego and social factors, enabling the model to handle unseen scene compositions. The combination of scene discretization, gating networks, and difficulty prediction constitutes a novel framework that significantly enhances zero-shot generalization capabilities.
Limitations
- The approach relies heavily on accurate feature extraction and scene discretization; errors in feature detection or clustering could impair scene factorization quality. Its effectiveness in highly dynamic or ambiguous scenarios remains to be validated.
- While the method reduces performance gaps in typical scenarios, extreme rare or complex scenarios may still pose challenges, necessitating further multimodal data integration and richer scene representations.
- Additional computational overhead from autoencoder training and gating mechanisms may hinder real-time deployment, requiring optimization for practical systems.
Future Work
Future research will focus on integrating multi-sensor data (LiDAR, camera, radar) to enrich scene features, further improving generalization. End-to-end training of scene factorization and trajectory prediction modules could reduce error propagation. Extending this framework to multi-agent, multi-task environments and real-time systems will be key to deploying safer autonomous vehicles in diverse traffic conditions.
AI Executive Summary
Autonomous driving systems rely heavily on trajectory prediction models to anticipate the future movements of surrounding traffic participants. However, these models often struggle with rare, safety-critical scenarios that are underrepresented in training data, leading to significant performance drops in out-of-distribution (OOD) conditions. Addressing this challenge requires not only better data collection but also more robust evaluation and generalization strategies.
This paper introduces LongComp, a novel framework that leverages scene factorization inspired by compositional zero-shot learning (CZSL). By decomposing traffic scenarios into discrete ego and social axes based on safety-relevant features, the authors construct challenging long-tail evaluation sets that simulate unseen scene combinations. Using autoencoders and k-means clustering, they discretize continuous scene features into meaningful categories, enabling the creation of both closed-world and open-world OOD test scenarios.
The core technical innovation involves integrating task-modular gating networks (TMN) and a difficulty prediction auxiliary head into baseline trajectory prediction models like Motion Transformer (MTR). TMNs dynamically adjust model responses based on scene context, while the auxiliary head predicts future prediction errors, guiding the model to focus on high-difficulty scenarios. Experimental results on the WOMD dataset show that, without these enhancements, the performance gap in OOD settings is 5.0% (closed) and 14.7% (open). After applying the proposed methods, these gaps shrink to 2.8% and 11.5%, respectively, demonstrating significantly improved robustness.
The significance of this work lies in its ability to improve autonomous driving safety by enabling models to better handle rare, unseen traffic situations. It provides a scalable, interpretable approach to scene understanding and generalization, with potential applications in multi-modal sensor fusion and real-time deployment. Despite these advances, challenges remain in scaling to highly dynamic scenarios and reducing computational costs, which will be addressed in future research. Overall, LongComp marks a substantial step toward safer, more reliable autonomous vehicles capable of navigating the unpredictable real world.
Deep Dive
Abstract
Methods for trajectory prediction in Autonomous Driving must contend with rare, safety-critical scenarios that make reliance on real-world data collection alone infeasible. To assess robustness under such conditions, we propose new long-tail evaluation settings that repartition datasets to create challenging out-of-distribution (OOD) test sets. We first introduce a safety-informed scenario factorization framework, which disentangles scenarios into discrete ego and social contexts. Building on analogies to compositional zero-shot image-labeling in Computer Vision, we then hold out novel context combinations to construct challenging closed-world and open-world settings. This process induces OOD performance gaps in future motion prediction of 5.0% and 14.7% in closed-world and open-world settings, respectively, relative to in-distribution performance for a state-of-the-art baseline. To improve generalization, we extend task-modular gating networks to operate within trajectory prediction models, and develop an auxiliary, difficulty-prediction head to refine internal representations. Our strategies jointly reduce the OOD performance gaps to 2.8% and 11.5% in the two settings, respectively, while still improving in-distribution performance.