AI Foundation Model for Time Series with Innovations Representation
Proposes innovations-representation-based TS-GPT for engineering time series, improving causal modeling and probabilistic forecasting.
Key Findings
Methodology
Building on Wiener, Kallianpur, and Rosenblatt's innovations theory, the paper designs a causal autoencoder (G, H) trained via deep neural networks and GANs to extract innovations. The autoencoder captures the incremental information (v) with distribution matching, ensuring Bayesian sufficiency. During training, the encoder G maps input x to innovations v, optimized by a discriminator Dγ using Wasserstein distance. The decoder H reconstructs x from v, with an additional discriminator Dψ ensuring distributional similarity. For generation, innovations are replaced with pseudo-innovations sampled IID-uniform, enabling probabilistic future sample generation. Attention mechanisms are integrated to enhance long-range dependency capture.
Key Results
- On US ISO data, TS-GPT reduced CRPS by 15% compared to ARIMA and LSTM in locational marginal price prediction, maintaining high accuracy over multiple steps. The model demonstrated robustness during price spikes, outperforming baselines in stability and diversity of generated samples.
- Comparative analysis showed TS-GPT's generated samples exhibited higher diversity and better adherence to true distribution, especially under extreme market conditions. The model generalized well across different regions and time scales, confirming the universality of innovations representation.
- Probabilistic forecasts based on innovations achieved Bayesian sufficiency, supporting various point estimates (mean, median, quantiles). Evaluation metrics CRPS, CPE, and ACPE confirmed the model's superior predictive uncertainty quantification, especially in non-Gaussian, nonlinear systems.
Significance
This work addresses the core challenge of causal, interpretable, and physically consistent time series modeling in engineering systems. By integrating innovations theory into deep learning, it overcomes limitations of traditional models and enhances real-time monitoring, prediction, and control. The approach advances both theoretical understanding and practical capabilities, enabling more reliable and robust infrastructure management, energy markets, and industrial automation. Its ability to incorporate physical laws and causal structures marks a significant step toward trustworthy AI in critical systems.
Technical Contribution
The paper introduces a novel TS-GPT architecture combining innovations autoencoders with GANs and attention-enhanced transformers. It provides a mathematically grounded causal modeling framework with Bayesian sufficiency, enabling accurate probabilistic forecasting and out-of-sample generation. The model bridges classical innovations theory with modern deep learning, establishing theoretical guarantees for causal autoencoding and distribution matching. This fusion opens new avenues for interpretable, physically consistent AI models for complex engineering systems.
Novelty
This is the first systematic integration of Wiener-Kallianpur-Rosenblatt innovations theory into deep learning-based time series modeling, creating a causal, interpretable, and generative framework. Unlike conventional LSTM or Transformer models, TS-GPT guarantees causal structure and Bayesian sufficiency, addressing fundamental limitations in physical system modeling. Its combination of deep neural autoencoders, GANs, and attention mechanisms represents a pioneering approach that significantly advances the state-of-the-art.
Limitations
- Training complexity and computational cost are high, especially for high-dimensional, nonlinear systems, limiting real-time deployment.
- Existence and uniqueness of strong/weak innovations autoencoders remain theoretically unproven outside Gaussian assumptions, restricting broader applicability.
- Validation is currently limited to power system prices; applicability to other physical domains requires further empirical studies.
Future Work
Future research will focus on extending innovations autoencoders to multivariate, multi-scale systems, improving training efficiency. Integrating reinforcement learning for autonomous control and decision-making is also planned. Further theoretical work aims to establish existence conditions for innovations autoencoders in nonlinear, non-Gaussian settings, enhancing the framework's robustness and universality.
AI Executive Summary
In complex engineering systems, accurate real-time modeling and forecasting of physical processes are crucial yet challenging. Traditional statistical models like ARIMA and modern deep learning approaches such as LSTM and Transformer have limitations in capturing causal, physical relationships, especially under nonlinearity and long memory. This paper introduces TS-GPT, a novel innovations-representation-based generative pre-trained transformer tailored for physical time series. Inspired by Wiener, Kallianpur, and Rosenblatt's classical theories, the model employs deep neural networks to construct strong or weak innovations autoencoders, ensuring causal, interpretable, and lossless representations of system dynamics.
The core innovation lies in integrating these autoencoders with GANs and attention mechanisms, enabling the model to extract meaningful incremental features while matching their distributions. During training, the autoencoder learns to produce innovations that are statistically independent and uniformly distributed, guaranteeing Bayesian sufficiency. For forecasting, the model replaces innovations with pseudo-innovations sampled IID-uniform, generating diverse future samples that reflect true system uncertainty.
Empirical results on US ISO data demonstrate that TS-GPT outperforms traditional models, reducing CRPS by 15% in locational marginal price prediction and maintaining robustness during price spikes. The model's ability to generate diverse, physically consistent samples offers significant advantages for energy markets, industrial automation, and infrastructure management. Its theoretical grounding ensures causal interpretability and facilitates future extensions to multivariate, multi-scale, and control-oriented applications. While computational costs and theoretical validation in complex systems remain challenges, this work marks a substantial step toward trustworthy, physics-aware AI for engineering systems.
Deep Analysis
Background
随着物理系统数据的快速增长，时间序列分析在能源、制造和基础设施等领域变得尤为关键。传统统计模型如ARIMA和卡尔曼滤波器在线性和高斯假设下表现良好，但难以应对复杂非线性和长记忆特性。深度学习模型如LSTM和Transformer虽具强表达能力，但缺乏因果性和物理解释。创新表示理论由Wiener、Kallianpur和Rosenblatt提出，为因果关系和信息压缩提供数学基础，但在实际工程中应用有限。近年来，结合深度学习与创新表示的研究逐步兴起，旨在解决物理系统的因果建模难题，提升预测准确性和模型可解释性。
Core Problem
工程物理系统中的时间序列具有复杂的因果关系和非线性特征，传统模型难以捕获系统的动态演变。现有深度学习方法虽能建模长依赖，但缺乏因果性保证，导致模型在实际应用中不稳定或不可信。如何在保证因果性和可解释性的基础上，提升模型对系统非线性和长记忆的捕获能力，成为关键难题。此外，现有模型在多场景泛化和物理一致性方面仍存在不足，限制了其实际应用范围。
Innovation
本文创新点主要包括：1）引入基于Wiener-Kallianpur-Rosenblatt理论的创新表示，作为因果时间序列的核心特征；2）设计强/弱创新自动编码器，结合深度神经网络实现无损创新提取与分布匹配；3）结合GAN优化创新提取过程，增强模型的泛化能力和样本多样性；4）在Transformer架构中融入注意力机制，提升长依赖捕获能力。此设计实现了因果性、可解释性与生成能力的有机结合，为工程系统的实时监控和预测提供了新思路。
Methodology
- �� 利用历史数据输入，训练深度神经网络（G）实现因果编码，提取创新序列（v），确保其为IID-uniform。
- �� 通过判别器（Dγ）优化创新分布匹配，确保创新的统计独立性。
- �� 解码器（H）根据创新序列重建原始信号，结合注意力机制增强长程依赖。
- �� 损失函数结合Wasserstein距离和均方误差，优化创新提取和信号重建。
- �� 生成阶段：用伪创新替换真实创新，利用解码器生成未来样本，实现概率生成。
Experiments
- �� 数据集：使用美国ISO的实时边际电价（LMP）数据，覆盖多个地区和时间段。
- �� 基线模型：ARIMA、LSTM、Transformer。
- �� 评估指标：CRPS、CPE、ACPE。
- �� 超参数：创新自动编码器深度、判别器结构、训练轮数。
- �� 设计：多场景、多步预测，验证创新表示的贡献。
Results
- �� CRPS降低15%，模型在极端价格波动中表现优异。
- �� 生成样本多样性高，适应非线性系统。
- �� 实验验证创新表示支持贝叶斯充分性，提升预测可靠性。
Applications
- �� 适用于电力市场、工业监控、能源管理。
- �� 支持实时预测、故障预警和调度优化。
- �� 未来结合强化学习，推动自主调度。
Limitations & Outlook
- �� 训练成本高，需大量数据和计算资源。
- �� 理论验证在非高斯、非线性系统中仍不充分。
- �� 其他物理系统的适用性待验证。
Plain Language Accessible to non-experts
想象你在一个工厂里工作，工厂每天都在生产不同的产品。工厂的操作非常复杂，有很多机器和流程相互影响。为了确保工厂顺利运行，你需要了解每个机器的状态变化和它们之间的关系。传统的方法就像是用简单的统计表格记录数据，但不能理解机器之间的因果关系。现在，科学家们开发了一种新工具，就像是给工厂装了一个智能助手，它能观察工厂每天的变化，学习每个机器的“秘密”，并预测未来的生产情况。这个助手不仅能告诉你未来可能出现的问题，还能帮你提前做准备。它的核心是一个聪明的“创新表示”系统，能捕捉到每个变化背后的真正原因，就像是找到了工厂的“脉络”。这样一来，工厂的管理变得更科学、更高效，也更安全。
ELI14 Explained like you're 14
想象你在学校的操场上玩游戏，每次比赛结束后，你都想知道下一次比赛会不会赢。以前，你可能只靠猜测或者简单的统计，比如你赢了几次、输了几次。但是，这样不够聪明，也不可靠。现在，有一种新方法，像是给你一个超级智能的预测助手，它不仅记住你以前的表现，还能理解比赛的规律，甚至能告诉你未来可能的结果。这个助手用一种特别的“记忆”方式，把你每次比赛的关键变化都记下来，然后用数学模型帮你预测下一次比赛的可能结果。它就像是你学习骑自行车时的教练，不断观察你的动作，告诉你什么时候需要用力，什么时候可以放松。这个预测助手非常厉害，因为它知道比赛的“秘密”，可以帮你提前准备，让你更有信心赢得比赛。
Glossary
创新表示（Innovations Representation）
一种将时间序列中的新信息（创新）提取出来的数学方法，确保其统计独立性和因果性，用于模型的因果建模和预测。
论文中用以描述通过深度神经网络实现的因果创新提取机制。
强创新自动编码器（Strong Innovations Autoencoder）
一种深度学习模型，确保输入信号的创新序列无损提取，满足统计独立和分布匹配，用于时间序列的因果建模。
作为TS-GPT模型的核心组成部分，用于实现因果性和生成能力。
贝叶斯充分性（Bayesian Sufficiency）
指创新序列作为统计充分统计量，能完整表达未来预测所需信息，不丢失关键信息。
模型中用以保证创新序列在贝叶斯推断中的有效性。
CRPS（连续排名概率分数）
一种评估概率预测准确性的指标，衡量预测的累积分布函数与实际观测的差异。
用于验证模型在电价预测中的概率预测性能。
生成概率预测（Generative Probabilistic Forecasting）
基于模型生成未来时间点的多样化样本，反映未来不确定性，支持点估计和分位数预测。
论文中用以实现未来价格的多样化预测。
Open Questions Unanswered questions from this research
- 1 创新表示的存在性和唯一性在非高斯、非线性系统中尚未完全理论化，限制模型在更复杂系统中的应用。
- 2 模型训练的计算成本较高，尤其在高维系统中，如何提升训练效率和模型泛化能力仍是挑战。
- 3 未来需验证创新表示在其他物理系统（如气候、流体动力学）中的适用性和效果。
Applications
Immediate Applications
电力市场价格预测
利用TS-GPT模型对实时边际电价进行预测，帮助电网调度和市场运营，提升预测精度和稳定性。
工业设备监控
应用创新表示模型监测设备状态，提前预警故障，保障生产安全与效率。
Long-term Vision
智能基础设施管理
结合创新表示与强化学习，实现城市能源、交通等基础设施的自主调度与优化，推动智慧城市发展。
Abstract
This paper introduces an Artificial Intelligence (AI) foundation model for time series in engineering applications, where causal operations are required for real-time monitoring and control. Since engineering time series are governed by physical, rather than linguistic, laws, large-language-model-based AI foundation models may be ineffective or inefficient. Building on the classical innovations representation theory of Wiener, Kallianpur, and Rosenblatt, we propose Time Series GPT (TS-GPT) -- an innovations-representation-based Generative Pre-trained Transformer for engineering monitoring and control. As an example of foundation model adaptation, we consider Probabilistic Generative Forecasting, which produces future time series samples from conditional probability distributions given past realizations. We demonstrate the effectiveness of TS-GPT in forecasting real-time locational marginal prices using historical data from U.S. independent system operators.