Numerical models outperform AI weather forecasts of record-breaking extremes
HRES numerical model outperforms AI models in forecasting record-breaking weather extremes.
Key Findings
Methodology
The study compares the HRES numerical model from the European Centre for Medium-Range Weather Forecasts with advanced AI models like GraphCast, Pangu-Weather, and Fuxi. It uses training data from 1979-2017 and tests these models on record-breaking weather events in 2018 and 2020.
Key Results
- HRES shows significantly lower errors than AI models in predicting record-breaking heat, cold, and wind events, especially in short-term forecasts where HRES's RMSE is noticeably lower.
- AI models systematically underestimate the frequency and intensity of record-breaking events, particularly for heat.
- AI models improve over longer lead times but still lag behind HRES.
Significance
The study underscores the importance of accurately predicting record-breaking weather events in a rapidly warming climate. While AI models excel in some tasks, they fall short in extreme event forecasting, which is crucial for disaster management and early warning systems.
Technical Contribution
The study provides a systematic error analysis of AI models in extreme weather event forecasting, revealing their limitations in extrapolating beyond the training domain. It establishes a benchmark for future model improvements.
Novelty
This is the first systematic evaluation of AI models' ability to forecast record-breaking weather events, revealing their shortcomings in out-of-distribution performance, offering new insights for AI model development.
Limitations
- AI models' errors in extreme event forecasting mainly stem from the limitations of training data, lacking sufficient exposure to extreme events.
- The study is limited to tests in 2018 and 2020, with a limited sample of years.
Future Work
Future research directions include improving AI models' extrapolation capabilities through data augmentation and hybrid modeling, particularly in extreme weather event forecasting.
AI Executive Summary
In recent years, AI models have made significant advances in weather forecasting, even surpassing traditional numerical weather prediction systems in some tasks. However, their ability to predict record-breaking extreme weather events remains unclear. This study compares the High RESolution forecast (HRES) numerical model from the European Centre for Medium-Range Weather Forecasts with advanced AI models like GraphCast, Pangu-Weather, and Fuxi, revealing AI models' shortcomings in predicting record-breaking heat, cold, and wind events. The study finds that AI models have significantly higher errors than HRES, especially in short-term forecasts. Despite AI models' advantages in speed and energy efficiency, they still require further improvement in extreme event forecasting. The study highlights the importance of accurately predicting record-breaking weather events in a rapidly warming climate and points out the current limitations of AI models in extrapolating beyond their training domain. Future research directions include improving AI models' extrapolation capabilities through data augmentation and hybrid modeling, particularly in extreme weather event forecasting.
Deep Analysis
Background
In recent years, AI models have made significant advances in weather forecasting, particularly excelling in medium to long-term predictions. However, predicting extreme weather events remains a challenge. Traditional numerical weather prediction systems like HRES rely on physical models and can predict extreme events to some extent, while AI models primarily rely on learning from historical data.
Core Problem
AI models perform worse than numerical models in predicting extreme weather events, especially record-breaking ones. Due to the rarity and extremity of these events, AI models rarely encounter similar situations during training, leading to errors in predictions.
Innovation
This study is the first to systematically evaluate AI models' ability to forecast record-breaking weather events, revealing their shortcomings in out-of-distribution performance. This provides new insights for further development of AI models.
Methodology
- �� Compare HRES and AI models in testing
- �� Construct a dataset of record-breaking events covering heat, cold, and wind in 2018 and 2020
- �� Evaluate model performance across different forecast lead times, focusing on short-term errors
Experiments
The experiments use historical data from 1979-2017 for training and test the models' predictive capabilities in 2018 and 2020. By comparing HRES and AI models' performance on record-breaking events, the study analyzes their errors and biases.
Results
Results show that HRES has significantly lower errors than AI models in predicting record-breaking heat, cold, and wind events, especially in short-term forecasts. AI models' errors mainly stem from underestimating the intensity and frequency of these events.
Applications
The study's findings are significant for disaster management and early warning systems, especially in a rapidly warming climate where accurately predicting extreme weather events is crucial.
Limitations & Outlook
AI models' errors in extreme event forecasting mainly stem from the limitations of training data, lacking sufficient exposure to extreme events. The study is limited to tests in 2018 and 2020, with a limited sample of years.
Plain Language Accessible to non-experts
Imagine weather forecasting as cooking. Traditional numerical weather prediction is like following a recipe, strictly adhering to steps and proportions to ensure the same dish is made each time. AI models are like an experienced chef, using past experiences and intuition to cook. While AI models excel in regular weather forecasting, they struggle with unprecedented extreme weather, like a chef encountering a new dish for the first time, leading to errors. The study finds that traditional numerical models are more accurate in predicting these extreme weather events because they rely on physical laws, like recipes, better handling unprecedented situations.
ELI14 Explained like you're 14
Weather forecasting is like playing a game. Sometimes you can predict your opponent's moves, but sometimes they surprise you. AI models are like a smart player who predicts weather based on past experiences, but when faced with unprecedented extreme weather, it's like encountering a new level, leading to mistakes. Traditional numerical models are like a player with a game guide, less flexible than AI models but more reliable in extreme weather. The study finds that numerical models are more accurate in predicting these extreme weather events because they rely on physical laws, like game guides, better handling unprecedented situations.
Glossary
HRES (High RESolution forecast)
A numerical weather prediction system relying on physical models, capable of predicting extreme events to some extent.
Used in the paper to compare AI models' predictive capabilities.
GraphCast
An advanced AI weather forecasting model using graph neural networks.
Used in the paper for comparison with HRES.
Pangu-Weather
An AI weather forecasting model using deep learning techniques for medium to long-term predictions.
Used in the paper for comparison with HRES.
Fuxi
An AI weather forecasting model using cascade machine learning techniques.
Used in the paper for comparison with HRES.
ERA5
A global climate reanalysis dataset providing detailed historical weather data.
Used in the paper for training and testing AI models.
Open Questions Unanswered questions from this research
- 1 AI models' errors in extreme weather event forecasting mainly stem from the limitations of training data, requiring more extreme event data for improvement.
- 2 Current AI models fall short in out-of-distribution performance, needing data augmentation and hybrid modeling for better extrapolation.
Applications
Immediate Applications
Disaster Warning Systems
Improving AI models' predictive capabilities to enhance the accuracy of disaster warning systems, reducing losses from extreme weather.
Climate Research
Using improved AI models for climate change research, predicting the frequency and intensity of future extreme weather events.
Long-term Vision
Global Climate Monitoring
Using improved AI models for global climate monitoring, providing more accurate climate change predictions to help formulate response strategies.
Abstract
Artificial intelligence (AI)-based models are revolutionizing weather forecasting and have surpassed leading numerical weather prediction systems on various benchmark tasks. However, their ability to extrapolate and reliably forecast unprecedented extreme events remains unclear. Here, we show that for record-breaking weather extremes, the numerical model High RESolution forecast (HRES) from the European Centre for Medium-Range Weather Forecasts still consistently outperforms state-of-the-art AI models GraphCast, GraphCast operational, Pangu-Weather, Pangu-Weather operational, and Fuxi. We demonstrate that forecast errors in AI models are consistently larger for record-breaking heat, cold, and wind than in HRES across nearly all lead times. We further find that the examined AI models tend to underestimate both the frequency and intensity of record-breaking events, and they underpredict hot records and overestimate cold records with growing errors for larger record exceedance. Our findings underscore the current limitations of AI weather models in extrapolating beyond their training domain and in forecasting the potentially most impactful record-breaking weather events that are particularly frequent in a rapidly warming climate. Further rigorous verification and model development is needed before these models can be solely relied upon for high-stakes applications such as early warning systems and disaster management.