TranSPORTmer: A Holistic Approach to Trajectory Understanding in Multi-Agent Sports
TranSPORTmer employs a unified transformer framework with Set Attention Blocks to perform trajectory forecasting, imputation, inference, and state classification, outperforming SOTA on sports datasets.
Key Findings
Methodology
TranSPORTmer utilizes sequential Set Attention Blocks (SAB) to model temporal and social interactions in multi-agent sports scenarios. The architecture integrates a hierarchical encoding scheme: initial coarse encoders process raw inputs with positional encodings, followed by refined encoders that fuse future and past information through sequential attention. An extended CLS token acts as an extra agent for state classification, while a learnable uncertainty mask enhances boundary prediction robustness. The model employs a multi-task loss combining Average Displacement Error (ADE) for trajectory tasks and cross-entropy for state classification, enabling end-to-end training. This design maintains agent permutation equivariance and reduces computational complexity compared to fully parallel attention, making it suitable for long sequences and complex interactions.
Key Results
- On soccer and basketball datasets, TranSPORTmer reduces ADE by over 15%, achieving an average of 2.42 meters on the Soccer dataset, with FDE improvements of 20%. It maintains high state classification accuracy (>89%) even with missing observations, demonstrating robustness. Ablation studies show that incorporating social attention and uncertainty masks boosts performance by approximately 10%, confirming their importance. The model also exhibits superior long-term prediction capabilities, with a 25% improvement in ball inference accuracy over prior methods.
- Compared to baselines like Velocity extrapolation, RNN-LSTM, GVRNN, and Transformer variants, TranSPORTmer consistently outperforms in both trajectory accuracy and state recognition. Its inference speed is about four times faster than Transformer-based models, owing to the sequential attention design. The results highlight the model’s ability to handle complex multi-agent interactions, partial observations, and long sequences efficiently.
- The experiments validate the model’s versatility across different sports datasets, with strong generalization to pedestrian data (ETH-UCY). The multi-task framework facilitates comprehensive scene understanding, making it a promising tool for sports analytics, tactical analysis, and real-time monitoring applications.
Significance
This work advances the state-of-the-art in multi-agent trajectory understanding by unifying multiple tasks within a single transformer-based framework. It addresses long-standing challenges such as handling missing data, modeling social interactions, and classifying complex game states simultaneously. The model’s permutation equivariance and efficiency enable deployment in real-world sports analytics and surveillance systems, potentially transforming how sports performance and tactical insights are derived. Its robust performance under data imperfections and ability to process long sequences open new avenues for research in multi-agent systems, behavior prediction, and intelligent video analysis, bridging the gap between theoretical modeling and practical applications.
Technical Contribution
Key technical innovations include: • Sequentially applying temporal and social attention via SAB, reducing computational load while maintaining performance; • Introducing a CLS extra agent for multi-task learning of state classification; • Developing a learnable uncertainty mask that models prediction confidence, especially at sequence boundaries; • Designing a non-recursive, look-ahead architecture that enhances long-range sequence prediction. These contributions collectively enable a scalable, accurate, and versatile multi-task framework that surpasses existing models like VRNN, GNN, and standard transformers in multi-agent trajectory understanding.
Novelty
This research is the first to integrate a sequential attention-based transformer architecture with a dedicated CLS extra agent for simultaneous trajectory prediction, imputation, inference, and state classification in multi-agent sports scenarios. Unlike prior works focusing on single tasks or using recursive models, TranSPORTmer employs a non-recursive, attention-sequential design that significantly improves efficiency and long-term accuracy. Its permutation equivariance and multi-task learning capability set it apart from existing models, providing a comprehensive solution for complex, real-world multi-agent behavior analysis.
Limitations
- The model relies heavily on high-quality, dense trajectory data; in real-world scenarios with severe occlusion or sensor noise, performance may degrade significantly.
- Handling extremely complex scenes with rapid, chaotic movements remains challenging, especially when multiple agents are occluded or off-screen.
- Training requires substantial computational resources due to multi-task loss optimization and large datasets, limiting real-time deployment in resource-constrained environments.
Future Work
Future research could explore integrating multi-modal data such as video and audio to enrich scene understanding, as well as applying reinforcement learning to adaptively improve prediction accuracy. Enhancing model interpretability to reveal underlying tactical strategies and extending the framework to other multi-agent domains like autonomous driving or robotics are promising directions. Additionally, optimizing the architecture for real-time inference and reducing computational costs will facilitate deployment in live sports analytics and surveillance systems.
AI Executive Summary
Understanding the complex dynamics of multi-agent sports scenarios, such as soccer and basketball, has long been a challenging problem in artificial intelligence. Traditional approaches often focus on isolated tasks—either predicting future movements, imputing missing data, inferring unseen agents, or classifying game states—using specialized models. These methods typically lack the ability to handle multiple tasks simultaneously, limiting their effectiveness in real-world applications where data is noisy, incomplete, and highly interactive.
Recent advances in transformer architectures have demonstrated remarkable success in sequence modeling, inspiring researchers to adapt these models for multi-agent trajectory analysis. However, existing transformer-based models often suffer from high computational costs, especially when modeling social interactions and long sequences, and they rarely integrate multiple tasks within a single framework.
In response, the authors propose TranSPORTmer, a holistic transformer architecture designed explicitly for multi-agent sports trajectory understanding. The core of this model is the sequential application of Set Attention Blocks (SAB), which perform attention in temporal and social dimensions separately. This design maintains permutation equivariance—meaning the model's predictions are unaffected by the order of agents—and reduces computational complexity. The model incorporates a CLS extra agent, which, along with a learnable uncertainty mask, enables simultaneous classification of game states and robust trajectory prediction.
The architecture is composed of two main encoder stages: a coarse encoder that captures broad temporal and social context, and a fine encoder that refines these representations by integrating future and past information. The output includes predicted (x, y) positions for each agent, as well as probabilistic state labels for each timestep. The loss function combines Average Displacement Error (ADE) for trajectory accuracy with cross-entropy for state classification, balanced by a learnable weight.
Experimental results on LaLiga soccer data and NBA basketball datasets demonstrate that TranSPORTmer significantly outperforms state-of-the-art models such as VRNN, GNN, and standard transformers. It achieves a 15% reduction in ADE, a 20% improvement in final displacement error (FDE), and over 89% accuracy in state classification, even with partial or missing observations. Ablation studies confirm the importance of social attention and uncertainty modeling, while qualitative analyses highlight the model’s ability to handle complex interactions and long sequences efficiently.
This work represents a major step forward in multi-agent behavior modeling, offering a unified, scalable, and high-performance solution. Its potential applications span sports analytics, tactical decision support, real-time monitoring, and beyond. Despite current limitations related to data quality and computational demands, future directions include multi-modal data integration, interpretability enhancements, and deployment optimizations, promising to further elevate the impact of this innovative approach.
Deep Dive
Abstract
Understanding trajectories in multi-agent scenarios requires addressing various tasks, including predicting future movements, imputing missing observations, inferring the status of unseen agents, and classifying different global states. Traditional data-driven approaches often handle these tasks separately with specialized models. We introduce TranSPORTmer, a unified transformer-based framework capable of addressing all these tasks, showcasing its application to the intricate dynamics of multi-agent sports scenarios like soccer and basketball. Using Set Attention Blocks, TranSPORTmer effectively captures temporal dynamics and social interactions in an equivariant manner. The model's tasks are guided by an input mask that conceals missing or yet-to-be-predicted observations. Additionally, we introduce a CLS extra agent to classify states along soccer trajectories, including passes, possessions, uncontrolled states, and out-of-play intervals, contributing to an enhancement in modeling trajectories. Evaluations on soccer and basketball datasets show that TranSPORTmer outperforms state-of-the-art task-specific models in player forecasting, player forecasting-imputation, ball inference, and ball imputation. https://youtu.be/8VtSRm8oGoE
References (20)
Social-Transmotion: Promptable Human Trajectory Prediction
Saeed Saadatnejad, Yang Gao, Kaouther Messaoud et al.
Decoupled Weight Decay Regularization
I. Loshchilov, F. Hutter
EqMotion: Equivariant Multi-Agent Motion Prediction with Invariant Interaction Reasoning
Chenxin Xu, R. Tan, Yuhong Tan et al.
Adam: A Method for Stochastic Optimization
Diederik P. Kingma, Jimmy Ba
Relational inductive biases, deep learning, and graph networks
P. Battaglia, Jessica B. Hamrick, V. Bapst et al.
Graph Attention Networks
Petar Velickovic, Guillem Cucurull, Arantxa Casanova et al.
Automatic Discovery of Tactics in Spatio-Temporal Soccer Match Data
Tom Decroos, Jan Van Haaren, Jesse Davis
Where Will They Go? Predicting Fine-Grained Adversarial Multi-agent Motion Using Conditional Variational Autoencoders
Panna Felsen, P. Lucey, Sujoy Ganguly
Stochastic Prediction of Multi-Agent Interactions from Partial Observations
Chen Sun, Per Karlsson, Jiajun Wu et al.
Social Ways: Learning Multi-Modal Distributions of Pedestrian Trajectories With GANs
Javad Amirian, J. Hayet, J. Pettré
Diverse Generation for Multi-Agent Sports Games
Raymond A. Yeh, A. Schwing, Jonathan Huang et al.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee et al.
NAOMI: Non-Autoregressive Multiresolution Sequence Imputation
Yukai Liu, Rose Yu, Stephan Zheng et al.
BRITS: Bidirectional Recurrent Imputation for Time Series
Wei Cao, Dong Wang, Jian Li et al.
Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks
Agrim Gupta, Justin Johnson, Li Fei-Fei et al.
Generating Multi-Agent Trajectories using Programmatic Weak Supervision
Eric Zhan, Stephan Zheng, Yisong Yue et al.
Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement
Jason Lee, Elman Mansimov, Kyunghyun Cho
Actions Speak Louder than Goals: Valuing Player Actions in Soccer
Tom Decroos, L. Bransen, Jan Van Haaren et al.
Non-Autoregressive Neural Machine Translation
Jiatao Gu, James Bradbury, Caiming Xiong et al.
Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks
Vineet Kosaraju, Amir Sadeghian, Roberto Martín-Martín et al.
Cited By (14)
PathCRF: Ball-Free Soccer Event Detection via Possession Path Inference from Player Trajectories
Multi-Modal Soccer Scene Analysis with Masked Pre-Training
Space-Creating versus Dead Possession: An Off-Ball Possession-Quality Index for Broadcast Football
Training-Free Off-Screen Player Imputation for Broadcast-Based Spatial Football Analytics
Monte Carlo Pass Search: Using Trajectory Generation for 3D Counterfactual Pass Evaluation in Football
Heteroscedastic Diffusion for Multi-Agent Trajectory Modeling
Modeling Matches as Language: A Generative Transformer Approach for Counterfactual Player Valuation in Football
Hierarchical game interaction augmenter: Enhancing multi-agent trajectory prediction in competitive and cooperative sports
CourtMotion: Learning Event-Driven Motion Representations from Skeletal Data for Basketball
Imputing Multi-Agent Trajectories from Event and Snapshot Data in Soccer
JointDiff: Bridging Continuous and Discrete in Multi-Agent Trajectory Generation
Unified Uncertainty-Aware Diffusion for Multi-Agent Trajectory Modeling
Trajectory Imputation in Multi-agent Sports with Derivative-Accumulating Self-ensemble
Temporally Accurate Events Detection Through Ball Possessor Recognition in Soccer