Scalable Memristive-Friendly Reservoir Computing for Time Series Classification

With the continuous development of deep learning, issues of computational efficiency, scalability, and energy consumption are becoming increasingly prominent. Particularly, the reliance on transformer-based large architectures has heightened the demand for alternative models that are both efficient and adaptable to emerging hardware constraints. Neuromorphic computing and reservoir computing, as frameworks that emulate the efficiency of biological or physical systems, have garnered significant interest in recent years. Echo State Networks (ESNs) in reservoir computing have been widely applied to time series classification tasks due to their lightweight and energy-efficient properties. However, traditional ESNs perform inefficiently on modern hardware accelerators, limiting their application in large-scale tasks. Memristors, as a new type of electronic component, are considered ideal building blocks for neuromorphic computing systems due to their nonlinear and memory characteristics. The memristive-friendly echo state network (MF-ESN) provides a low-power machine learning solution by simulating memristive dynamics.

Traditional ESNs and MF-ESNs perform inefficiently on modern hardware accelerators, limiting their application in large-scale tasks. Although the memristive-friendly echo state network (MF-ESN) provides a low-power machine learning solution by simulating memristive dynamics, its inherent sequential nature makes it difficult to parallelize on modern hardware. This limitation has contributed to the dominance of transformers, whose attention mechanism is highly parallelizable. To overcome this challenge, this paper proposes a novel architecture called MARS (Memristive-Friendly Parallelized Reservoir), which achieves efficient scalable parallel computation and deeper model composition through innovative subtractive skip connections.

The core innovations of the MARS model are:

• �� Introduction of subtractive skip connections: By incorporating subtractive skip connections into the model, MARS achieves deeper model composition, enhancing the model's expressiveness.

• �� Parallel scan algorithm: The parallel scan algorithm optimizes forward computation, significantly enhancing training speed and predictive performance.

• �� Retention of memristive dynamics: MARS retains memristive dynamics while achieving parallelization, ensuring the model's low-power characteristics.

These innovations enable MARS to significantly outperform strong gradient-based models like LRU, S5, and Mamba on multiple long sequence benchmarks, achieving a 21x training speedup under the same hardware conditions.

The implementation of the MARS model includes the following key steps:

• �� Initialization: Randomly initialize input matrices and memristor parameters to ensure model diversity and stability.

• �� Parallelization: Use the parallel scan algorithm to optimize forward computation, significantly enhancing training speed and predictive performance.

• �� Subtractive skip connections: Incorporate subtractive skip connections into the model to achieve deeper model composition, enhancing the model's expressiveness.

• �� Retention of memristive dynamics: Retain memristive dynamics while achieving parallelization, ensuring the model's low-power characteristics.

• �� Output layer optimization: Optimize the output weight tensor through global linear regression or classification to ensure model prediction accuracy.

The experimental design includes two main scenarios. First, we evaluate the scaling behavior and classification ability of MARS compared to classical ESN and MF-ESN. The experimental setup aims to assess improvements over previous versions. Second, we test MARS on several real-world classification benchmarks against heavyweight gradient-based SoTA models to demonstrate that efficiency does not mean poor performance. The experiments use datasets like UCR and UEA-MTSCA, covering time series tasks of varying lengths and complexities. We also conducted ablation studies to verify the contribution of each component in the model.

Experimental results show that MARS significantly outperforms strong gradient-based models like LRU, S5, and Mamba on multiple long sequence benchmarks, reducing full training time from minutes or hours to seconds or even just a few hundred milliseconds. MARS achieves 100% accuracy on the Coffee task and 99.6% on the Wafer task from the UCR dataset. Additionally, MARS achieves a 21x training speedup under the same hardware conditions. Ablation studies indicate that subtractive skip connections and the parallel scan algorithm play key roles in enhancing model performance.

The MARS model has broad application potential in multiple fields. Direct application scenarios include real-time signal processing and embedded systems, especially in situations requiring efficient low-power computation. MARS's efficiency and scalability make it advantageous in handling large-scale time series data, applicable in fields like financial market analysis and weather forecasting. Additionally, MARS's design provides a clear pathway to energy-efficient, low-latency implementations on emerging memristive and in-memory hardware.

Despite MARS's outstanding performance in multiple tasks, it may experience information loss when handling extremely long sequences, as the memristive dynamic state may rapidly enter a specific growth or decay state. Additionally, the model's performance may not match the most advanced gradient models in certain tasks, especially those requiring high adaptability. MARS's hyperparameter selection currently relies on manual tuning, which may affect optimal performance. Future research directions include developing automated hyperparameter optimization strategies to further enhance MARS's performance.