State Propagation Also Satisfies: A Complex-Valued State-Space Model for Deterministic State Tracking

Sequence modeling has evolved from RNNs to Transformer architectures, which leverage attention for capturing long-range dependencies. While attention mechanisms excel in many tasks, their quadratic complexity and parameter overhead limit scalability, especially for long sequences. Linear state-space models like S4 and Mamba have been proposed to reduce complexity, but they primarily excel in continuous or memory-based tasks, not in discrete, exact memorization. Recent advances suggest complex-valued representations can enhance expressiveness, especially for cyclic or periodic patterns. This background motivates exploring models that combine complex domain properties with minimalistic recurrence for precise discrete state tracking.

Existing sequence models struggle with tasks requiring exact memorization of discrete states, such as parity or parenthesis matching. Attention mechanisms, though powerful, are computationally intensive and over-parameterized for simple deterministic functions. Linear models often cannot represent exact cycles due to eigenvalue restrictions. Deep RNNs face gradient issues, and current linear SSMs lack the capacity for precise discrete state transitions. The core challenge is designing a lightweight, stable model capable of exact cyclic state representation, scalable to deep architectures, and trainable without gradient vanishing or explosion.

This work introduces CSP, a complex-valued state-space model that relies solely on state propagation. Key innovations include: 1) element-wise learned phase rotations enabling exact cyclic state transitions; 2) elimination of intermediate output projections, simplifying the architecture; 3) block-level skip connections and complex normalization for deep stability; 4) phase-focused decoding that encodes information in the angular domain. These innovations collectively enable the model to perform precise discrete memory tasks efficiently, outperforming traditional RNNs and linear models in accuracy and parameter efficiency.

• �� Input sequences are linearly transformed into complex vectors at each timestep. • For each input, a rotation angle θt is computed via a learnable projection, controlling phase rotation. • The input vector is rotated element-wise by eiθt, updating the phase. • The rotated input combines with the previous state through a complex recurrence, modulated by learned decay αt and scale γt. • Block-level skip connections add the original input after nonlinear activation, preserving information. • Complex normalization projects the state onto the unit circle, emphasizing phase encoding. • Multiple layers stack these operations, with training stabilized via Adam optimizer, learning rate decay, and gradient clipping.

The model was tested on three tasks: parity check, mod-3 counting, and parenthesis matching, each with sequence length 16. Using 3 layers, 64 hidden units, and training over 300 epochs, CSP achieved perfect accuracy and F1 scores. Ablation studies confirmed the importance of rotation, skip connections, and complex normalization. The Focal Loss was employed in imbalanced scenarios, notably for parenthesis matching, ensuring effective learning. The training dynamics exhibited grokking, with long plateau phases followed by sudden convergence, indicating phase boundary crossing. Comparisons with Transformer and linear models demonstrated superior efficiency and exactness in discrete tasks.

CSP achieved 100% accuracy and F1 scores across all tasks, with convergence in fewer than 150 epochs. Ablation experiments showed that removing rotation or normalization caused performance collapse, confirming their critical roles. The use of Focal Loss was essential in imbalanced class scenarios. The training process revealed a grokking phenomenon, where models remained near chance for many epochs before rapidly achieving perfect generalization, driven by phase boundary crossing in the complex domain. These results validate CSP’s capacity for exact discrete state tracking with high efficiency.

CSP is suitable for symbolic reasoning, logic verification, and tasks requiring exact memorization, such as code analysis or formal language processing. Its parameter efficiency makes it ideal for resource-constrained environments like edge devices. The approach can be extended to natural language tasks involving cyclic or repetitive patterns, and integrated into hardware-efficient AI systems for embedded applications, enabling fast, accurate, and low-cost sequence processing.

The current model is primarily validated on synthetic, discrete tasks; its performance on noisy, real-world data remains untested. Sensitivity to phase boundary handling may affect robustness. Scalability to very long sequences or multi-modal data is uncertain. The decoding mechanism involving phase extraction adds parameter overhead, which future work aims to optimize. Further research is needed to adapt CSP to continuous, noisy, or high-dimensional data scenarios.