A Critical Audit of Spatiotemporal Forecasting Benchmark Datasets and Baselines
Classical time series methods reveal linear models outperform GNNs on benchmarks, questioning dataset discriminative power.
Key Findings
Methodology
This paper employs classical time series tools—Pearson correlation, Ljung-Box test, partial autocorrelation—to analyze five widely used datasets (Chickenpox, PedalMe, WikiMaths, METR-LA, PEMS-BAY). It examines spatial and temporal dependencies, assesses the impact of differencing, and compares linear models like SARIMA with deep GNNs. The analysis reveals data biases, especially overdifferencing, which inflates linear model performance. Additionally, residual learning combining SARIMA errors with GNNs is tested to improve predictions. The approach emphasizes rigorous statistical evaluation to understand data structure and model effectiveness.
Key Results
- Analysis shows Chickenpox and PedalMe datasets exhibit overdifferencing, leading linear models to outperform complex GNNs, with SARIMA achieving lower MSE on WikiMaths. Traffic datasets (METR-LA, PEMS-BAY) display long memory and strong spatial correlations, but these diminish after differencing. Unprocessed data reveals that simple models like SARIMA and residual learning outperform deep models in certain scenarios, highlighting the importance of data preprocessing. Residual SARIMA-GNN models significantly improve traffic prediction accuracy, surpassing previous SOTA. Overall, the study demonstrates that traditional statistical models remain competitive, especially on biased datasets.
- The findings challenge the current reliance on complex deep models for benchmarking, emphasizing the need for more representative datasets and rigorous evaluation protocols. The results suggest that in many real-world scenarios, simpler models leveraging classical statistics can achieve comparable or superior performance, especially when data biases are present. This insight encourages a re-evaluation of benchmarking practices and promotes hybrid approaches combining traditional models with deep learning for robust, generalizable predictions.
Significance
This work critically re-examines the evaluation standards in spatiotemporal forecasting, exposing biases in widely used datasets that favor linear models. It underscores the importance of understanding data structure and advocates for more rigorous statistical validation. The demonstration that simple models like SARIMA can outperform complex GNNs on certain benchmarks questions the current paradigm, urging the community to develop more challenging, realistic datasets. The residual learning approach exemplifies how traditional statistical methods can complement deep learning, opening new avenues for robust, interpretable models. Overall, the study pushes for a more scientific, data-aware approach in the development and assessment of forecasting models, with implications for both research and industry applications.
Technical Contribution
This paper introduces a systematic statistical analysis framework for evaluating spatiotemporal datasets, revealing biases such as overdifferencing and weak spatial signals. It demonstrates the effectiveness of classical models like SARIMA as strong baselines, and innovatively combines SARIMA residuals with GNNs to enhance predictive performance. The approach emphasizes the importance of data preprocessing and rigorous evaluation, providing a new perspective on model comparison. The integration of traditional time series models with deep learning architectures offers a hybrid methodology that balances interpretability and accuracy, setting a new standard for benchmarking in the field.
Novelty
This is the first comprehensive statistical analysis of widely used spatiotemporal datasets, uncovering biases caused by overdifferencing and data preprocessing. It proposes using SARIMA residuals as training targets for GNNs, a novel hybrid approach that outperforms existing models on traffic datasets. The work challenges the prevailing focus on complex deep models, highlighting the enduring relevance of classical statistical methods. Its insights into data structure and evaluation protocols provide a new lens for future research, emphasizing data quality and statistical rigor over mere model complexity.
Limitations
- The analysis primarily relies on linear correlation measures, which may overlook complex nonlinear dependencies present in real data.
- The datasets studied are limited in scope; results may vary with different data sources or more heterogeneous scenarios.
- Residual learning strategies, while effective, may not generalize well to highly noisy or non-stationary environments, requiring further validation.
Future Work
Future research should incorporate nonlinear dependency measures, such as mutual information, to deepen understanding of data structures. Developing more diverse, realistic benchmark datasets that reflect complex real-world scenarios is crucial. Exploring multi-modal data fusion, adaptive models, and real-time deployment will enhance practical applicability. Additionally, extending residual learning strategies and hybrid models to other domains like finance or climate prediction could broaden impact. Emphasizing data quality and statistical validation will be key to advancing robust, interpretable forecasting methods.
AI Executive Summary
Spatiotemporal forecasting has become a cornerstone of modern data science, with graph neural networks (GNNs) leading the charge in modeling complex spatial dependencies. However, despite the proliferation of sophisticated models, the evaluation landscape remains dominated by a handful of benchmark datasets, including Chickenpox, PedalMe, WikiMaths, METR-LA, and PEMS-BAY. These datasets, sourced from diverse real-world scenarios such as public health, web activity, and traffic, have been instrumental in advancing the field. Yet, their statistical properties and preprocessing methods, particularly differencing, have not been critically examined. This oversight has led to surprising findings: linear models, especially those uninformed by spatial structure, often outperform or rival deep GNNs on these benchmarks. The authors employ classical time series tools—Pearson correlation, partial autocorrelation, Ljung-Box test—to analyze these datasets, revealing pervasive overdifferencing and weak spatial signals. These biases artificially inflate the performance of simple models, casting doubt on the datasets' discriminative power.
Building on this insight, the paper advocates for a more rigorous evaluation framework. It demonstrates that models like SARIMA, a traditional linear time series method, serve as strong baselines, especially when trained on raw, undifferenced data. Moreover, the authors introduce a hybrid residual learning approach, combining SARIMA errors with GNNs, which significantly enhances prediction accuracy in traffic datasets, surpassing existing state-of-the-art methods. These results underscore the importance of understanding data structure and preprocessing effects, urging the community to revisit benchmarking practices.
Overall, this work challenges the prevailing narrative that deep models are inherently superior in spatiotemporal forecasting. It highlights that classical models, when properly tuned and evaluated, remain powerful tools. The findings call for the development of more representative datasets and evaluation protocols rooted in statistical rigor, ensuring that future models are truly robust and generalizable. By integrating traditional statistical insights with modern deep learning, the paper paves the way for more reliable, interpretable, and effective forecasting solutions across various domains.
Deep Dive
Abstract
Graph neural networks (GNNs) are routinely employed for short-range forecasting on multivariate time series with a spatial graph structure. Despite the availability of many alternative datasets, method innovations within this domain are predominantly assessed against a rather limited set of benchmark datasets, most notably Chickenpox, PedalMe, WikiMaths, METR-LA, and PEMS-BAY. The evaluation protocols contain baselines spanning from historical averages to classical machine learning approaches. These baselines often show competitive performance compared to GNNs. In the present work, we take a step back and analyse the benchmark datasets via classical time series methods to uncover why spatially-unaware linear models pose a stronger competitor than previously reported, casting further doubt on the discriminative reliability of the aforementioned widely adopted datasets. Our statistical analysis provides a toolset for identifying significant spatial and temporal correlations, while revealing a structural bias introduced by first-order differenced datasets. We therefore recommend reducing the over-reliance on such datasets for method comparison, and instead advocate for more rigorous statistical evaluation. By applying the results of our analysis to a simple hybrid model, we show how our methodology can lead to novel ways of developing GNN models