Lightweight Spatio-Temporal Attention Network with Graph Embedding and Rotational Position Encoding for Traffic Forecasting
Proposed LSTAN-GERPE leverages multi-layer spatio-temporal attention, RoPE, and graph eigenvector embedding to significantly improve traffic forecasting accuracy.
Key Findings
Methodology
The LSTAN-GERPE model employs a lightweight multi-layer spatio-temporal self-attention framework, separately capturing spatial and temporal dependencies through dedicated attention modules. It introduces Rotary Position Encoding (RoPE) with systematically optimized frequencies to enhance sequence dependency learning. The model further incorporates road network topology by performing eigenvalue decomposition on the normalized graph Laplacian, embedding structural information into the feature space. The architecture consists of a data embedding block, multiple stacked spatio-temporal attention layers, and a final output layer, trained with Huber loss to balance robustness and sensitivity. This design avoids extensive feature engineering, relying instead on intrinsic attention mechanisms and graph structural cues, leading to superior performance on real-world datasets PeMS04 and PeMS08.
Key Results
- On PeMS04, the model achieved an MAE of 18.65 and RMSE of 30.53, outperforming baseline models such as DCRNN (MAE 21.92, RMSE 35.01) and Traffic-Transformer (MAE 18.92, RMSE 31.34), reducing errors by approximately 10-15%.
- On PeMS08, the MAE was 14.90 and RMSE 24.33, demonstrating consistent superiority across datasets with different scales and traffic patterns.
- Ablation studies revealed that removing RoPE increased MAE by about 1 point, and excluding spatial attention led to significant performance drops, confirming the importance of these components. The integration of graph eigenvector features contributed to further error reduction, validating the effectiveness of the combined approach.
Significance
This work advances traffic forecasting by effectively modeling long-range spatial dependencies and complex dynamic patterns, overcoming the limitations of short-range GNNs and high-parameter Transformer models. Its lightweight design makes it suitable for real-time deployment in urban traffic management systems, facilitating smarter routing, congestion mitigation, and environmental benefits. The integration of graph topology and attention mechanisms paves the way for more robust, scalable, and interpretable models, addressing critical challenges in intelligent transportation systems and urban planning.
Technical Contribution
The primary technical innovations include: 1) a multi-layer lightweight spatio-temporal attention architecture that separately models spatial and temporal dependencies; 2) the systematic optimization of RoPE frequency parameters via grid search, enhancing long- and short-term sequence modeling; 3) the embedding of road network structural features through eigenvalue decomposition of the normalized Laplacian, enriching spatial feature representation; 4) a parameter-efficient design that balances model complexity and performance, enabling deployment in resource-constrained environments. These contributions collectively push the frontier of traffic prediction models, combining theoretical insights with practical efficiency.
Novelty
This study is the first to systematically incorporate RoPE with frequency tuning into traffic forecasting models, effectively capturing long-range dependencies without excessive parameter overhead. It uniquely combines graph spectral features with attention mechanisms, bridging the gap between structural graph information and sequence modeling. Unlike prior works relying heavily on feature engineering or complex transformers, this approach emphasizes a lightweight, end-to-end architecture that maintains high accuracy while reducing computational costs. The integration of eigenvector-based spatial features into attention modules marks a novel contribution to spatio-temporal modeling.
Limitations
- The model's performance under extreme traffic conditions, such as accidents or weather disruptions, remains untested, as training data lacks sufficient representation of such scenarios.
- Dependence on high-quality road network data means that inaccuracies or missing information can impair prediction accuracy.
- While lightweight compared to full Transformer models, the computational cost may still be significant for ultra-large-scale networks, requiring further optimization for real-time applications.
Future Work
Future directions include integrating multi-source data such as weather, events, and social media signals to improve robustness. Developing adaptive frequency tuning for RoPE could enhance model flexibility across diverse scenarios. Extending the framework to multi-task learning, such as combining traffic flow prediction with anomaly detection, offers promising avenues. Additionally, exploring edge deployment and real-time inference optimization will be crucial for practical urban traffic management solutions.
AI Executive Summary
Urban traffic congestion remains a pressing challenge worldwide, demanding more accurate and scalable prediction methods to optimize flow and reduce environmental impact. Traditional statistical models like ARIMA and Kalman filters provided initial solutions but struggled with complex, nonlinear, and long-range dependencies inherent in city traffic data. The advent of deep learning, particularly graph neural networks (GNNs) and Transformer-based models, marked a significant leap forward. Yet, these models often suffer from high computational complexity or limited capacity to model long-distance spatial dependencies.
This paper introduces the Lightweight Spatio-Temporal Attention Network with Graph Embedding and Rotational Position Encoding (LSTAN-GERPE), a novel approach designed to address these limitations. The core idea is to leverage multi-layered attention mechanisms that separately focus on spatial and temporal dependencies, enabling the model to learn intricate traffic patterns over long distances and time horizons. Unlike traditional GNNs that primarily consider local neighborhood information, LSTAN-GERPE captures long-range dependencies through a combination of attention and spectral graph features.
A key innovation is the integration of Rotary Position Encoding (RoPE), which encodes positional information with a tunable frequency parameter. The authors systematically optimize this parameter via grid search, ensuring the model effectively captures both short-term and long-term sequence dependencies. Additionally, the model incorporates the structural features of the road network by performing eigenvalue decomposition on the normalized graph Laplacian, embedding topological information directly into the feature space.
Experimental results on the PeMS04 and PeMS08 datasets demonstrate the model’s superior accuracy, with MAE reductions of over 10% compared to state-of-the-art baselines. Ablation studies confirm that each component—RoPE, spatial attention, and graph spectral features—contributes significantly to performance gains. The lightweight design ensures practical deployability, making it suitable for real-time traffic management systems.
Overall, this research offers a comprehensive solution that combines advanced attention mechanisms, spectral graph theory, and systematic parameter tuning to push the boundaries of traffic forecasting. It provides a scalable, interpretable, and efficient framework that can be extended to other spatio-temporal domains, paving the way for smarter, more responsive urban transportation systems. Despite its strengths, the model’s robustness under extreme conditions and reliance on high-quality data remain areas for future exploration. Continued development along these lines promises to further enhance urban mobility and sustainability.
Deep Dive
Abstract
Traffic forecasting is a key task in the field of Intelligent Transportation Systems. Recent research on traffic forecasting has mainly focused on combining graph neural networks (GNNs) with other models. However, GNNs only consider short-range spatial information. In this study, we present a novel model termed LSTAN-GERPE (Lightweight Spatio-Temporal Attention Network with Graph Embedding and Rotational Position Encoding). This model leverages both Temporal and Spatial Attention mechanisms to effectively capture long-range traffic dynamics. Additionally, the optimal frequency for rotational position encoding is determined through a grid search approach in both the spatial and temporal attention mechanisms. This systematic optimization enables the model to effectively capture complex traffic patterns. The model also enhances feature representation by incorporating geographical location maps into the spatio-temporal embeddings. Without extensive feature engineering, the proposed method in this paper achieves advanced accuracy on the real-world traffic forecasting datasets PeMS04 and PeMS08.
References (20)
Attention is All you Need
Ashish Vaswani, Noam Shazeer, Niki Parmar et al.
Spatio-temporal Graph Convolutional Neural Network: A Deep Learning Framework for Traffic Forecasting
Ting Yu, Haoteng Yin, Zhanxing Zhu
Spatial-Temporal Graph ODE Networks for Traffic Flow Forecasting
Zheng Fang, Qingqing Long, Guojie Song et al.
A Comprehensive Survey on Graph Neural Networks
Zonghan Wu, Shirui Pan, Fengwen Chen et al.
Integrating Granger Causality and Vector Auto-Regression for Traffic Prediction of Large-Scale WLANs
Zheng Lu, Chen Zhou, Jing Wu et al.
Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting
Yaguang Li, Rose Yu, C. Shahabi et al.
Self-Attention with Relative Position Representations
Peter Shaw, Jakob Uszkoreit, Ashish Vaswani
A Novel Spatio-Temporal Model for City-Scale Traffic Speed Prediction
Kun Niu, Huiyang Zhang, Tong Zhou et al.
Spatial-Temporal Synchronous Graph Convolutional Networks: A New Framework for Spatial-Temporal Network Data Forecasting
Chao Song, Youfang Lin, S. Guo et al.
Learning Dynamic and Hierarchical Traffic Spatiotemporal Features With Transformer
Haoyang Yan, Xiaolei Ma, Ziyuan Pu
RoFormer: Enhanced Transformer with Rotary Position Embedding
Jianlin Su, Yu Lu, Shengfeng Pan et al.
LLGformer: Learnable Long-range Graph Transformer for Traffic Flow Prediction
Di Jin, Cuiying Huo, Jiayi Shi et al.
PDFormer: Propagation Delay-aware Dynamic Long-range Transformer for Traffic Flow Prediction
Jiawei Jiang, Chengkai Han, Wayne Xin Zhao et al.
Spatio-Temporal Adaptive Embedding Makes Vanilla Transformer SOTA for Traffic Forecasting
Hangchen Liu, Zheng Dong, Renhe Jiang et al.
A novel hybrid time-varying graph neural network for traffic flow forecasting
Ben-Ao Dai, Bao-Lin Ye
Evaluations of Multi-Step Traffic Flow Prediction Models Based on Graph Neural Networks
Yaopeng Xie, Cheng‐Jie Jin
Adaptive Graph Convolutional Recurrent Network with Transformer and Whale Optimization Algorithm for Traffic Flow Prediction
Chen Zhang, Yue Wu, Ya Shen et al.
Enhancing Spatial-Temporal Awareness via Graph Convolutional Networks and Transformers for Traffic Flow Forecasting
Daoming Lu
High-Performance Spatio-Temporal Information Mixer for Traffic Forecasting
Yuanpei Huang, Nanfeng Xiao
A Hybrid Transformer-Based Spatial-Temporal Network for Traffic Flow Prediction
Guanqun Tian, Dequan Li
Cited By (2)
HLSR: Hybrid Live Forecast Selective Dynamic Vehicle Rerouting for Real-Time Congestion Avoidance
A novel adaptive spatio-temporal dual-branch model for traffic flow prediction