SEMPO: Lightweight Foundation Models for Time Series Forecasting
SEMPO is a lightweight foundation model for time series forecasting, combining energy-aware spectral decomposition and prompt-based Transformer.
Key Findings
Methodology
SEMPO integrates Energy-aware Spectral Decomposition (EASD) and Mixture-of-Prompts Transformer (MoPFormer). EASD adaptively partitions the spectrum based on energy thresholds, enhancing low-energy signal modeling via multi-band spectral masks. MoPFormer employs small, dataset-specific prompts routed dynamically to experts, improving parameter efficiency and cross-domain adaptation. Pretraining uses energy-aware reconstruction on multi-domain data, with prompt tuning during fine-tuning. The combined architecture reduces model size and data needs while maintaining strong generalization.
Key Results
- On 16 datasets, SEMPO outperforms state-of-the-art models in zero- and few-shot scenarios, with only 6.5M parameters pre-trained on 83M points, reducing errors by 12% (zero-shot) and 22% (few-shot).
- In benchmarks like ETTh1 and ETTm2, it surpasses existing models in MSE and MAE, demonstrating robust cross-domain transfer.
- Ablation studies confirm that spectral partitioning boosts low-energy frequency modeling, and prompt routing enhances heterogeneity handling, with combined effects leading to superior performance.
Significance
This work addresses the heavy reliance of current models on large data and parameters, proposing a resource-efficient alternative that retains high accuracy. Its spectral decomposition enhances understanding of complex temporal dynamics, enabling deployment in industrial, energy, and climate applications. The significant reduction in model size and training data paves the way for democratizing advanced time series forecasting, especially in resource-constrained environments.
Technical Contribution
The paper introduces a novel spectral partitioning mechanism that adaptively emphasizes low-energy signals, improving transferability. The MoPFormer architecture leverages small, dynamic prompts routed via a learned gating mechanism, reducing parameter count while capturing diverse patterns. The combination of these innovations results in a theoretically sound and practically efficient framework that outperforms large-scale models in both accuracy and resource utilization.
Novelty
This is the first work to incorporate energy-aware spectral decomposition into time series pretraining, explicitly addressing the bias towards high-energy frequencies. The integration of a small prompt-based routing mechanism within Transformer layers offers a new lightweight approach to modeling heterogeneous temporal data, setting it apart from prior large-scale mixture-of-experts models. The dual focus on spectral utilization and parameter efficiency marks a significant innovation.
Limitations
- The spectral decomposition relies on signal quality; in noisy or sparse frequency scenarios, performance may degrade. The adaptive thresholds may need further tuning for extreme cases.
- Pretraining on diverse datasets helps generalization, but domain gaps can still limit transfer effectiveness in highly divergent target domains.
- While parameter-efficient, the model may still face challenges in ultra-high-frequency or real-time applications, requiring further optimization.
Future Work
Future directions include developing more robust spectral partitioning strategies, integrating multi-modal data, and exploring self-supervised learning to further enhance low-energy spectrum modeling. Additionally, optimizing the model for real-time deployment and edge devices will expand its practical impact. Extending the framework to multivariate and high-frequency data streams is also a promising avenue.
AI Executive Summary
The rapid growth of large pre-trained models has revolutionized time series forecasting, yet their enormous size and data requirements limit deployment in resource-constrained environments. Hui He et al. address this challenge by proposing SEMPO, a lightweight yet powerful foundation model that combines energy-aware spectral decomposition with a prompt-based Transformer architecture.
Traditional models tend to focus on high-energy frequency components, neglecting subtle but informative low-energy signals critical for capturing complex temporal dynamics. SEMPO’s EASD module adaptively partitions the spectrum based on learned energy thresholds, applying multi-band spectral masks to emphasize low-energy frequencies. This approach enriches the model’s understanding of persistent, subtle patterns often overlooked.
Complementing this, the MoPFormer module introduces a small set of task-specific prompts, dynamically routed via an adaptive gating mechanism to specialized experts within the Transformer. This design allows the model to efficiently handle heterogeneous data from diverse domains without the need for massive parameters, significantly reducing computational costs.
Extensive experiments on 16 datasets demonstrate SEMPO’s superiority in zero- and few-shot forecasting scenarios. With only 6.5 million parameters trained on 83 million data points, it outperforms models with hundreds of millions of parameters, reducing forecasting errors by an average of 12% and 22%. These results highlight the model’s strong generalization and transferability, making it highly suitable for industrial applications where resources are limited.
The research advances the field by showing that intelligent spectral analysis and prompt routing can achieve high performance with minimal resources. Future work will focus on enhancing spectral robustness, integrating multi-modal data, and deploying in real-time settings, promising a new era of accessible, efficient time series forecasting.
Deep Analysis
Background
时间序列预测在工业自动化、能源管理和气候监测等领域扮演关键角色。早期采用统计模型如指数平滑和ARIMA，逐步过渡到深度学习架构如Transformer，显著提升了预测精度。近年来，预训练基础模型如Moirai和Chronos通过多域学习实现迁移，但其庞大参数规模带来高昂的计算成本，限制了实际应用。轻量化模型的研究逐渐兴起，试图在保持性能的同时减小模型规模，但仍面临数据利用不足和跨域适应性差的问题。
Core Problem
现有模型普遍依赖大规模数据和庞大网络，导致训练和推理成本高昂，难以在资源有限环境中部署。模型偏向高能量频谱，忽略微弱但信息丰富的低能量频段，影响泛化能力。如何在保证预测性能的同时，显著减小模型参数和预训练数据，是当前的核心难题。解决方案需兼顾频谱利用效率和模型参数高效性，以实现跨域迁移和少样本学习。
Innovation
本文提出两大创新：第一，EASD模块通过自适应能量阈值，将频谱划分为高低能量部分，增强低能量频段的建模能力，改善模型对微弱信号的捕获。第二，MoPFormer引入小规模任务提示专家池，结合动态路由机制，提升模型对异质时间序列的适应性，降低参数规模。这两者结合实现了模型参数和预训练数据的双重压缩，突破了传统大模型的限制。
Methodology
- �� 频谱分解：将时间序列经过FFT变换，利用自适应能量阈值将频谱划分为高低能量部分。
- �� 多频段掩码：在高低能量频谱中，生成多个频段掩码，选择性抑制特定频率范围。
- �� 逆FFT还原：频谱掩码后，利用逆FFT还原时域信号，增强低能量频谱的表达。
- �� Patchify与投影：将频谱处理后的信号划分为不重叠块，线性投影为Transformer输入。
- �� MoP机制：引入任务特定提示，通过动态路由，将不同时间序列tokens分配到不同专家，提升模型适应性。
- �� Transformer架构：采用多层堆叠的MoPFormer块，结合标准自注意力机制和提示融合，进行序列建模。
- �� 预训练：采用能量感知的重建目标，利用多域数据进行自监督训练。
- �� 微调：冻结Transformer骨架，仅调优提示和预测头，适应特定任务。
Experiments
使用涵盖能源、气候、交通等多个领域的83M时间点数据进行预训练，评估在TSLib和GIFT-Eval基准上的零样本和少样本预测性能。对比模型包括大规模预训练模型（如Time-MoE、Chronos）和轻量模型（如Moirai、TimesFM）。指标主要为MSE和MAE，重点考察模型在不同预测步长和跨域迁移中的表现。还进行了消融实验验证频谱分解和提示机制的贡献。
Abstract
The recent boom of large pre-trained models witnesses remarkable success in developing foundation models (FMs) for time series forecasting. Despite impressive performance across diverse downstream forecasting tasks, existing time series FMs possess massive network architectures and require substantial pre-training on large-scale datasets, which significantly hinders their deployment in resource-constrained environments. In response to this growing tension between versatility and affordability, we propose SEMPO, a novel lightweight foundation model that requires pretraining on relatively small-scale data, yet exhibits strong general time series forecasting. Concretely, SEMPO comprises two key modules: 1) energy-aware SpEctral decomposition module, that substantially improves the utilization of pre-training data by modeling not only the high-energy frequency signals but also the low-energy yet informative frequency signals that are ignored in current methods; and 2) Mixture-of-PrOmpts enabled Transformer, that learns heterogeneous temporal patterns through small dataset-specific prompts and adaptively routes time series tokens to prompt-based experts for parameter-efficient model adaptation across different datasets and domains. Equipped with these modules, SEMPO significantly reduces both pre-training data scale and model size, while achieving strong generalization. Extensive experiments on two large-scale benchmarks covering 16 datasets demonstrate the superior performance of SEMPO in both zero-shot and few-shot forecasting scenarios compared with state-of-the-art methods. Code and data are available at https://github.com/mala-lab/SEMPO.