One Fits All:Power General Time Series Analysis by Pretrained LM
FPT freezes GPT-2 blocks for seven time-series tasks, reaching 0.516 average long-horizon MSE and 74.0% classification accuracy.
Key Findings
Methodology
Frozen Pretrained Transformer (FPT) preserves and freezes GPT-2 self-attention and feed-forward networks, training only input and positional embeddings, LayerNorm, and task heads. Time series are normalized with RevIN, grouped into local Patching tokens, and linearly projected into GPT-2's hidden dimension. The same backbone supports forecasting, imputation, classification, and anomaly detection.
Key Results
- Across Weather, Traffic, Electricity, ILI, and four ETT datasets, GPT2(6) obtains average long-term forecasting MSE 0.516, a 9.3% relative reduction from TimesNet's 0.596.
- On ten UEA classification datasets, GPT2(6) reaches 74.00% average accuracy versus TimesNet's 73.60%. On SMD, MSL, SMAP, SWaT, and PSM, it obtains 86.72% average F1 versus 85.24% for TimesNet.
- For imputation, GPT2(3) averages MSE/MAE 0.047/0.127, reducing average MSE by 4.1% versus TimesNet. With only 10% training data, average forecasting MSE is 0.371, 33.3% below TimesNet.
Significance
The paper addresses a central obstacle to time-series foundation models: unlike NLP, the field lacks massive pretraining corpora, with the largest datasets below 10GB. Reusing pretrained language or vision models offers a unified, low-label route for energy, industrial, traffic, and healthcare data. The results suggest that transferability is not restricted to semantic language knowledge; generic sequence-processing behavior can also be valuable for numerical dynamics.
Technical Contribution
FPT separates modality adaptation from generic computation. RevIN mitigates instance-level distribution shifts, Patching increases effective history while controlling token count, and linear probing maps numerical patches into the pretrained representation space. Freezing attention and FFN reduces trainable parameters and catastrophic forgetting. The paper further provides theoretical and empirical evidence that self-attention behaves similarly to PCA, offering a mechanism for cross-domain universality.
Novelty
Compared with TimesNet, PatchTST, DLinear, and other time-series-specific architectures, this work is the first systematic demonstration that one largely frozen pretrained Transformer can cover classification, anomaly detection, imputation, short- and long-term forecasting, few-shot learning, and zero-shot transfer. Its conceptual novelty is linking cross-modal transfer to the PCA-like behavior of self-attention.
Limitations
- GPT-2 was not pretrained on temporal signals; performance therefore depends on the learned input adapter, normalization, patch size, and task head. Frozen features may underfit abrupt regime changes, causal mechanisms, or highly domain-specific periodicity.
- Zero-shot experiments involve a limited set of benchmark datasets and cannot establish reliability in finance, clinical monitoring, or strongly non-stationary control systems. The paper also does not fully characterize latency, memory use, or uncertainty calibration.
Future Work
Future work should build larger and more diverse temporal corpora, investigate selective unfreezing and time-series-specific pretraining, and study variable-aware representations. BERT and BEiT should be evaluated more broadly, while efficient long-context attention, online adaptation, calibrated uncertainty, and robust transfer under distribution shift remain important engineering priorities.
AI Executive Summary
Forecasting, classification, imputation, and anomaly detection are central to energy, industry, transport, and medicine, yet time-series research remains fragmented: each task typically receives a specialized model. The contrast with NLP is stark. Public time-series data are small—under 10GB for the largest datasets—making it difficult to train a foundation model from scratch. One Fits All asks whether an existing language model can provide the missing computational substrate.
Its answer is Frozen Pretrained Transformer (FPT). Using GPT-2 as the main backbone, the method freezes self-attention and feed-forward blocks while training input embeddings, positional embeddings, LayerNorm, and task-specific outputs. RevIN removes instance-level scale differences; Patching converts neighboring observations into compact tokens. The authors argue that self-attention performs operations resembling PCA, allowing a model trained on language to extract broadly useful patterns from numerical sequences. Experiments with BERT and the vision model BEiT further support architectural and cross-modal universality.
The empirical case is broad. GPT2(6) achieves average long-term forecasting MSE 0.516 versus 0.596 for TimesNet, a 9.3% relative reduction. It reaches 74.00% accuracy on ten UEA classification datasets and 86.72% average F1 on five anomaly benchmarks. GPT2(3) obtains average imputation MSE 0.047, while 10% few-shot forecasting reaches MSE 0.371, 33.3% below TimesNet. Zero-shot transfer gives an average metric of 16.38, outperforming DLinear, TimesNet, and PatchTST. The broader lesson is not that GPT-2 universally dominates, but that a frozen cross-modal Transformer can function as a reusable computational engine. Its main caveats are limited benchmark diversity, dependence on adaptation layers, and uncertain robustness under extreme non-stationarity.
Deep Analysis
Background
Transformers have driven Informer, Autoformer, FEDformer, PatchTST, and TimesNet, but these models are generally designed for particular temporal tasks. In NLP, GPT-2 and BERT gain transferability from large corpora; in vision, BEiT demonstrates token-based reuse. Time-series data are far smaller than language data, so direct foundation-model pretraining is difficult. Cross-modal reuse is therefore a practical alternative.
Core Problem
The central question is whether language or vision parameters can process numerical sequences without redesigning the Transformer core. The mismatch involves token statistics, variable interactions, scale variation, missing values, long horizons, and distribution shift. A successful solution must also avoid overfitting and catastrophic forgetting when downstream datasets are small.
Innovation
- �� FPT freezes GPT-2 attention and FFN blocks across seven tasks.
- �� RevIN and Patching provide numerical-to-token adaptation.
- �� GPT2(3) and GPT2(6) test depth selection.
- �� Theory and experiments connect attention to PCA-like projection.
- �� BERT and BEiT experiments test universality across architectures and modalities.
Methodology
- �� Input: multivariate windows, commonly length 96 or longer.
- �� Normalization: RevIN removes instance mean and scale, then restores them at output.
- �� Tokenization: Patching aggregates adjacent observations, reducing redundancy and increasing effective context.
- �� Embedding: a linear layer maps patches to GPT-2's hidden dimension.
- �� Backbone: multi-head self-attention and FFN remain frozen; positional embeddings and LayerNorm are fine-tuned.
- �� Head: separate outputs produce forecasts, reconstructions, class labels, or anomaly scores.
- �� Evaluation: anomaly detection uses reconstruction error for fairness.
Experiments
Datasets include ETTh1, ETTh2, ETTm1, ETTm2, Electricity, Weather, Traffic, ILI, M4, UEA, and SMD, MSL, SMAP, SWaT, PSM. Baselines include TimesNet, PatchTST, DLinear, FEDformer, N-BEATS, and Anomaly Transformer. Metrics are MSE, MAE, accuracy, F1, sMAPE, MASE, and OWA. Tests cover 12.5–50% masking, 10% and 5% few-shot training, cross-dataset zero-shot transfer, and ablations with random initialization, no pretraining, and unfrozen blocks.
Results
GPT2(6) obtains long-term forecasting MSE/MAE 0.516/0.407 versus TimesNet's 0.596/0.433. Classification accuracy is 74.00% versus 73.60%; anomaly F1 is 86.72% versus 85.24%. GPT2(3) reaches imputation MSE 0.047. On M4, sMAPE is 11.991, close to TimesNet's 11.829. Zero-shot average error is 16.38 versus PatchTST's 17.67. Ablations show that pretraining and partial freezing both improve results.
Applications
Energy providers can forecast Electricity demand; weather and traffic platforms can support planning and warnings; industrial operators can monitor SMD-, SWaT-, or PSM-like sensor streams. Healthcare, retail, and IoT users with limited labels may benefit from few-shot transfer. Deployment still requires calibration of windows, normalization, missing-data handling, and alert thresholds.
Limitations & Outlook
FPT is not a fully pretrained temporal foundation model: input adapters and output heads still require task data. Freezing lowers training cost but can limit adaptation to abrupt regimes, causal mechanisms, and extreme distribution shifts. Public benchmarks are relatively small and narrow, while long contexts retain Transformer memory and computation costs. Future systems should combine temporal pretraining, parameter-efficient unfreezing, uncertainty estimation, and online adaptation.
Plain Language Accessible to non-experts
Imagine a highly experienced craftsperson who learned to recognize patterns while working in a language factory. The craftsperson does not know every product, but has learned a general skill: compare many items, notice which ones matter, and track how they relate over time. FPT sends electricity readings, temperatures, or traffic counts through a translator, called the input adapter. RevIN first puts measurements from different machines onto a comparable ruler. Patching bundles nearby readings into small cards, making the long stream easier to handle. The craftsperson's central reasoning machinery stays locked; only the entrance and exit are adjusted. The same worker can then predict the next readings, fill missing ones, classify a sequence, or flag something unusual. In the paper, long-horizon average error is 0.516 versus 0.596 for TimesNet. Yet the craftsperson is not magically an expert in every industry. Sudden regime changes, new sensors, or unusual failures may require more retraining.
ELI14 Explained like you're 14
Picture a gaming champion who has spent years learning a strategy game. The champion never played the “predict electricity” game, but learned useful habits: watch what happened earlier, compare different clues, and focus on the most important moves. Now you give the champion temperature, traffic, or power numbers. Instead of teaching everything from zero, you turn the numbers into small cards and translate them into a format the champion understands.
That is FPT. It uses GPT-2's trained brain, but locks most of it. Only the entrance and exit are adjusted. RevIN is like putting every player's score on the same scale, while Patching combines nearby moments into one card. The exit can answer different questions: “What happens next?”, “What was missing?”, “Which category is this?”, or “Is this strange?”
The results are impressive: long-term forecasting MSE is 0.516 versus TimesNet's 0.596; classification accuracy is 74.00%; anomaly F1 is 86.72%. With only 10% of the training data, average error is 33.3% lower than TimesNet. Old skills really can help with a new game!
But it is not magic. GPT-2 never specifically learned factory failures or weather storms. If the rules suddenly change, it may struggle. Future models could study more time-series data and learn to update safely while working.
Glossary
Frozen Pretrained Transformer (FPT)
A Transformer that freezes most pretrained parameters and trains only lightweight adapters and task heads. This reduces training cost and limits catastrophic forgetting.
The paper uses FPT as one backbone for seven time-series tasks.
Self-Attention
A mechanism that weights positions according to their relationships and aggregates sequence information. The paper argues that its behavior can resemble PCA-like projection.
GPT-2 self-attention is retained and frozen.
Patching
Grouping adjacent observations into a local segment token. It reduces token redundancy while preserving local temporal structure.
FPT applies Patching after instance normalization.
RevIN
Reversible Instance Normalization removes sample-specific mean and scale, then restores them after prediction. It is designed to mitigate distribution shift.
RevIN adapts numerical inputs to the pretrained backbone.
Few-shot and zero-shot learning
Few-shot learning uses only a small amount of target data; zero-shot learning uses no target training data. Both measure transferability.
The paper evaluates 10% and 5% training regimes and cross-dataset zero-shot forecasting.
Open Questions Unanswered questions from this research
- 1 The PCA interpretation of attention remains incomplete: the roles of individual layers, heads, and positional embeddings in temporal computation are not fully identified.
- 2 Benchmark coverage is limited; robustness in noisy, causal, and strongly non-stationary finance or clinical data remains uncertain.
- 3 The paper does not fully quantify long-context memory, latency, energy cost, or predictive uncertainty.
Applications
Immediate Applications
Industrial anomaly monitoring
Operators can feed sensor streams resembling SMD, SWaT, or PSM into FPT and use reconstruction error for alerts. Deployment requires historical windows, mostly normal data, normalization calibration, and threshold selection.
Energy and traffic forecasting
Utilities and mobility platforms can fine-tune the output layer on Electricity or Traffic data to predict demand and flow. Before deployment, they should validate sampling frequency, missingness, seasonality, and distribution drift.
Long-term Vision
General-purpose temporal foundation model
A broader temporal corpus could let one model share representations across forecasting, imputation, classification, and monitoring, reducing repeated model development across industries. Major obstacles are data governance, drift, and reliable uncertainty.
Abstract
Although we have witnessed great success of pre-trained models in natural language processing (NLP) and computer vision (CV), limited progress has been made for general time series analysis. Unlike NLP and CV where a unified model can be used to perform different tasks, specially designed approach still dominates in each time series analysis task such as classification, anomaly detection, forecasting, and few-shot learning. The main challenge that blocks the development of pre-trained model for time series analysis is the lack of a large amount of data for training. In this work, we address this challenge by leveraging language or CV models, pre-trained from billions of tokens, for time series analysis. Specifically, we refrain from altering the self-attention and feedforward layers of the residual blocks in the pre-trained language or image model. This model, known as the Frozen Pretrained Transformer (FPT), is evaluated through fine-tuning on all major types of tasks involving time series. Our results demonstrate that pre-trained models on natural language or images can lead to a comparable or state-of-the-art performance in all main time series analysis tasks, as illustrated in Figure 1. We also found both theoretically and empirically that the self-attention module behaviors similarly to principle component analysis (PCA), an observation that helps explains how transformer bridges the domain gap and a crucial step towards understanding the universality of a pre-trained transformer.The code is publicly available at https://github.com/DAMO-DI-ML/One_Fits_All.