Parameter Efficient Hybrid Transformer (PEHT) for Network Traffic Prediction via Dynamic Urban Congestion Integration
PEHT integrates LoRA-enhanced Transformer with urban mobility and congestion data, achieving state-of-the-art network traffic prediction with over 90% parameter reduction.
Key Findings
Methodology
This paper introduces the PEHT framework, which separates primary network communication features from secondary urban mobility and congestion data. It employs a customized Transformer encoder embedded with Low-Rank Adaptation (LoRA) to drastically reduce trainable parameters—over 90%—while maintaining high prediction accuracy. The model uses a multimodal fusion strategy, injecting external mobility and congestion features into the decoder to enhance forecasting. The process involves: • Clustering spatial grid data into Virtual Base Stations (VBS) via a greedy algorithm that maximizes a utility function balancing spatial proximity, load similarity, and temporal correlation; • Constructing multivariate feature vectors combining historical traffic, temporal, static, and dynamic features, embedded via learnable layers; • Applying LoRA during training to approximate weight updates with low-rank matrices, reducing parameters significantly; • Fusing external features with encoder outputs in the decoder stage, ensuring temporal causality; • Employing an Encoder-Decoder Transformer architecture with multi-head self-attention to model long-range dependencies, producing future traffic predictions. Extensive experiments on Telecom Italia Milan dataset and synthetic congestion scenarios demonstrate superior performance over existing models in RMSE, MAE, and R-squared metrics.
Key Results
- On the Milan dataset, PEHT achieved RMSE values of 18.42, 34.85, and 122.10 for SMS, Call, and Internet traffic respectively, representing reductions of 14.6%, 11.9%, and 11.4% compared to the best baseline ST2T. MAE scores were 12.15, 20.60, and 68.45, with $R^2$ exceeding 0.95, confirming high predictive accuracy across multiple traffic types;
- Across five synthetic congestion scenarios, PEHT consistently outperformed baselines with the lowest RMSE (as low as 8.85), demonstrating robustness under diverse mobility and congestion conditions. Ablation studies revealed that integrating LoRA reduced parameters from 19.1 million to under 140,000 while maintaining performance, highlighting the model’s parameter efficiency.
- The ablation experiments confirmed that LoRA's parameter reduction did not compromise accuracy, and the fusion of external mobility features significantly enhanced the model’s responsiveness to traffic fluctuations. The results underscore the model’s scalability and practical deployment potential in large-scale urban networks.
Significance
This work addresses a critical bottleneck in urban network traffic prediction—parameter explosion in Transformer models—by introducing a parameter-efficient approach that leverages external urban mobility and congestion data. It advances the state-of-the-art by demonstrating that high-accuracy predictions are achievable with drastically fewer trainable parameters, facilitating deployment in resource-constrained environments. The integration of external signals directly influences network management strategies, enabling proactive resource allocation, congestion mitigation, and quality of service improvements. This approach aligns with the broader goal of developing intelligent, adaptive urban communication infrastructures, essential for the future of smart cities and 6G networks.
Technical Contribution
The core technical innovation lies in embedding LoRA into the Transformer encoder, which approximates large weight matrices with low-rank products, reducing parameter count by over 90%. This is combined with a multimodal fusion strategy that injects external mobility and congestion features into the decoder, maintaining temporal causality. The virtual base station clustering algorithm enhances data robustness by aggregating spatially adjacent grids based on a utility function that considers correlation, distance, and load variance. The architecture effectively captures long-range dependencies via multi-head self-attention, while the parameter-efficient design ensures scalability and computational feasibility for large urban datasets. These innovations collectively push the boundary of network traffic prediction models, making them more practical for real-world deployment.
Novelty
This study is the first to incorporate LoRA into Transformer-based urban network traffic prediction, achieving over 90% parameter reduction without sacrificing accuracy. It innovatively combines external urban mobility and congestion signals in a multimodal fusion framework, ensuring temporal causality and robustness. The virtual base station clustering algorithm further distinguishes this work by addressing data sparsity and spatial heterogeneity. Compared to prior models like ST2T and HGCRN, PEHT offers a unique blend of parameter efficiency, external data integration, and long-range dependency modeling, filling critical gaps in existing literature.
Limitations
- While the model performs well under typical urban conditions, its robustness during extreme events such as accidents or sudden infrastructure failures remains limited; future work should incorporate real-time sensor data and anomaly detection mechanisms.
- The current fusion strategy assumes static external features; dynamic, real-time external data streams like weather or event schedules could further improve accuracy but are not yet integrated.
- Despite significant parameter reduction, the model’s inference speed in real-time scenarios needs further optimization, especially for deployment on edge devices with limited computational resources.
Future Work
Future research will focus on integrating more diverse real-time external data sources, such as weather and event data, to improve model responsiveness. Additionally, developing adaptive fusion mechanisms that dynamically weigh external signals based on context could enhance robustness. Exploring federated learning frameworks will enable collaborative model training across multiple regions without data sharing, further scaling deployment. Finally, optimizing inference speed through model pruning and hardware-aware design will facilitate real-time deployment in resource-constrained environments, advancing the practical application of intelligent urban network management.
AI Executive Summary
Urban cellular networks face increasing demands for accurate traffic prediction to optimize resource allocation and maintain service quality. Traditional models, often based on statistical or simple deep learning approaches, struggle to capture the complex, dynamic nature of urban traffic influenced by diverse mobility patterns and congestion phenomena. Recent advances in Graph Neural Networks and Transformer architectures have improved spatio-temporal modeling capabilities; however, these methods often suffer from high computational costs and parameter explosion, limiting their scalability and real-world applicability.
This paper introduces the Parameter-Efficient Hybrid Transformer (PEHT), a novel framework designed to address these challenges. The core innovation is the integration of Low-Rank Adaptation (LoRA) into the Transformer encoder, drastically reducing the number of trainable parameters—by over 90%—while preserving, or even enhancing, predictive accuracy. The model distinguishes itself by effectively incorporating external urban mobility and congestion data through a multimodal fusion strategy, injecting these signals into the decoder stage to improve forecast reliability.
The architecture begins with a spatial clustering process, where raw grid data are aggregated into Virtual Base Stations (VBS) via a greedy utility-maximizing algorithm. This step mitigates data sparsity and spatial heterogeneity, creating more robust regional signals. The features—comprising historical traffic, temporal, static, and dynamic network data—are embedded into high-dimensional vectors. The Transformer encoder, enhanced with LoRA, processes these inputs, capturing long-range dependencies crucial for accurate forecasting.
In the decoding phase, external mobility and congestion features are fused with the encoder output, ensuring the model maintains strict temporal causality. The multi-head self-attention mechanism enables the model to understand complex spatial-temporal relationships, producing future traffic predictions with high precision. Extensive experiments on the Telecom Italia Milan dataset and synthetic congestion scenarios demonstrate that PEHT outperforms existing state-of-the-art models, reducing RMSE by approximately 14.6% and MAE by 11.9% in real-world data, while maintaining robustness across diverse traffic conditions.
The significance of this work lies in its ability to deliver high-accuracy predictions with drastically fewer parameters, making large-scale deployment feasible in urban environments. It bridges the gap between external urban signals and network traffic modeling, providing a pathway toward smarter, more adaptive network management systems. Future directions include integrating more real-time external data, optimizing inference for edge deployment, and exploring federated learning to enable collaborative, privacy-preserving model training across multiple cities. Despite current limitations in handling extreme events and real-time data streams, this research marks a substantial step forward in intelligent urban network traffic forecasting, with broad implications for smart city infrastructure and next-generation wireless networks.
Deep Dive
Abstract
Accurate network traffic prediction is a critical element for efficient resource allocation in dynamic urban cellular networks. However, prediction remains challenging because network demand is influenced by complex mobility patterns, congestion dynamics, and heterogeneous user behavior. This paper introduces the Parameter-Efficient Hybrid Transformer (PEHT), a network traffic prediction framework that integrates urban mobility and congestion information into a Transformer-based architecture. PEHT separates primary network communication features from secondary urban mobility features and incorporates Low-Rank Adaptation (LoRA) into the Transformer encoder to reduce the number of trainable parameters while maintaining high predictive accuracy. A multimodal fusion strategy then injects external mobility and congestion features into the decoder to improve traffic forecasting. Experiments on the Telecom Italia Milan dataset and multiple synthetic congestion scenarios show that PEHT outperforms state-of-the-art baselines in terms of RMSE, MAE, and $R^2$. The implementation is available in the GitHub repository.