SCORE: Replacing Layer Stacking with Contractive Recurrent Depth
SCORE replaces layer stacking with recurrent shared blocks inspired by ODEs, improving training speed and reducing parameters in GNNs, MLPs, and Transformers.
Key Findings
Methodology
This paper introduces SCORE (Skip-Connection ODE Recurrent Embedding), which replaces traditional deep layer stacks with iterative application of a single shared neural block, inspired by ODEs. The core update ht+1 = (1 - Δt) * ht + Δt * Fθ(ht) leverages fixed-step Euler discretization, avoiding complex ODE solvers. By sharing weights across iterations, the approach models depth as a dynamic evolution, enhancing stability and efficiency. Extensive experiments across GNNs, MLPs, and Transformers demonstrate faster convergence, parameter reduction, and comparable or improved performance, especially in low-data regimes.
Key Results
- On the ESOL molecular solubility dataset, SCORE-based GNNs achieved a mean RMSE of 0.533, outperforming traditional stacking models. In nanoGPT experiments, the recurrent Transformer with SCORE converged faster, with validation loss improved by 0.26 and parameters reduced by 20%. Euler integrator provided the best trade-off between computational cost and accuracy, with four steps yielding marginal gains over simpler methods. Across architectures, SCORE variants consistently ranked among top performers, demonstrating robustness and versatility.
- In dense neural networks, SCORE-MLP matched baseline accuracy while reducing variance and parameters, indicating enhanced stability. In graph models, SCORE variants outperformed several baselines, including GAT, GINE, and GCN, with the best RMSE around 0.533. In language models, the fixed-step recurrent approach improved training speed and model compactness, validating its broad applicability.
- Comparison of numerical integrators revealed Euler's method as optimal in practice, balancing simplicity and performance. Parameter sharing introduced implicit regularization, beneficial in low-data scenarios. The approach's flexibility allows adaptation to various architectures, making it a promising direction for lightweight, stable deep models.
Significance
This work fundamentally shifts the paradigm of deep neural network design by replacing layer stacking with recurrent, ODE-inspired updates. It addresses longstanding issues of parameter bloat, training instability, and over-smoothing, offering a lightweight, scalable alternative. The parameter sharing and fixed-step discretization simplify training, reduce computational costs, and enhance stability, especially in resource-constrained settings. Its successful application across GNNs, MLPs, and Transformers underscores its broad impact, paving the way for more efficient, interpretable, and robust deep learning architectures. This innovation has the potential to accelerate the deployment of large models and facilitate research in low-resource environments.
Technical Contribution
The core technical innovation lies in reformulating deep architectures as recurrent applications of a single neural block governed by an ODE-inspired contraction mapping. The fixed-step Euler discretization provides a simple yet effective numerical scheme, avoiding the complexity of adaptive solvers. Weight sharing across iterations introduces implicit regularization, improving generalization. The approach unifies diverse architectures—GNNs, MLPs, Transformers—under a common recurrent framework, demonstrating versatility and robustness. Theoretical insights into stability and contraction properties underpin the empirical success, offering a new perspective on deep model design.
Novelty
This is the first work to embed ODE-inspired contraction mappings directly into recurrent neural updates as a replacement for layer stacking, bypassing the need for continuous solvers. Unlike neural ODEs, which rely on adaptive integrators, SCORE employs fixed-step Euler updates with shared weights, simplifying implementation and training. The explicit control of update magnitude via Δt offers a novel stability mechanism. This combination of recurrence, weight sharing, and ODE principles constitutes a significant innovation in deep learning architecture design.
Limitations
- While effective in many scenarios, the fixed-step approach may face challenges in extremely deep or highly nonlinear tasks, where adaptive step sizing could be beneficial. Stability might degrade if Δt is not properly tuned, especially in complex architectures.
- The method's reliance on fixed iteration counts limits flexibility; dynamic or data-dependent iteration schemes could further improve performance but are unexplored.
- In very large-scale models, the recursive application may increase computational overhead, necessitating further optimization or hybrid schemes to balance efficiency and accuracy.
Future Work
Future research could develop adaptive step size strategies, integrating learnable or data-driven Δt schedules to enhance stability and performance. Extending SCORE to multi-task and transfer learning scenarios, as well as combining it with sparsity or quantization techniques, could further reduce model size and inference latency. Exploring higher-order integrators or stochastic variants may improve accuracy in complex tasks. Additionally, theoretical analysis of convergence and stability in broader settings will strengthen the framework's foundation.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在做一道菜，传统的方法是每次加入不同的调料，每次都要重新准备。这样既繁琐又容易出错。现在，假设你用一种特别的调料，只需要用同一瓶，反复搅拌几次，就能让菜变得更香、更均匀。这瓶调料代表共享的神经块，反复用同一份，既省材料，又能让菜更好吃。SCORE就像这个厨房技巧，用简单的反复操作取代复杂的多层堆叠，既快又稳。它让深度学习模型变得更轻、更快，还能保持效果，就像厨师用这招做出美味佳肴一样。
Abstract
Residual connections are central to modern deep neural networks, enabling stable optimization and efficient information flow across depth. In this work, we propose SCORE (Skip-Connection ODE Recurrent Embedding), a discrete recurrent alternative to classical layer stacking. Instead of composing multiple independent layers, SCORE iteratively applies a single shared neural block using an ODE (Ordinary Differential Equation)-inspired contractive update: ht+1 = (1 - dt) * ht + dt * F(ht) This formulation can be interpreted as a depth-by-iteration refinement process, where the step size dt explicitly controls stability and update magnitude. Unlike continuous Neural ODE approaches, SCORE uses a fixed number of discrete iterations and standard backpropagation without requiring ODE solvers or adjoint methods. We evaluate SCORE across graph neural networks (ESOL molecular solubility), multilayer perceptrons, and Transformer-based language models (nanoGPT). Across architectures, SCORE generally improves convergence speed and often accelerates training. SCORE is reducing parameter count through shared weights. In practice, simple Euler integration provides the best trade-off between computational cost and performance, while higher-order integrators yield marginal gains at increased compute. These results suggest that controlled recurrent depth with contractive residual updates offers a lightweight and effective alternative to classical stacking in deep neural networks.