Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning
Introducing SiLU/dSiLU activations combined with on-policy TD(λ)/Sarsa(λ), surpassing DQN in Atari and Tetris benchmarks.
Key Findings
Methodology
This study introduces Sigmoid-weighted Linear Units (SiLU) and their derivatives (dSiLU) as novel activation functions for neural network function approximation. These are integrated with on-policy TD(λ) and Sarsa(λ) algorithms, eliminating the need for experience replay and target networks. Experiments on SZ-Tetris, 10×10 Tetris, and Atari 2600 demonstrate that networks utilizing SiLU/dSiLU outperform traditional ReLU and sigmoid units. The approach employs softmax action selection with temperature annealing, optimizing exploration-exploitation balance. The core mechanism involves multiplying sigmoid functions with inputs to produce activations, which stabilize learning and improve value estimation accuracy across various tasks.
Key Results
- In stochastic SZ-Tetris, shallow dSiLU networks improved average scores by 20%, setting new state-of-the-art benchmarks. Deep networks with SiLU in convolutional layers and dSiLU in fully connected layers achieved an average of 4,900 points in 10×10 Tetris, surpassing previous bests. In Atari 2600, SiLU-dSiLU deep networks outperformed DQN, with an average score increase of 232% across 12 games, demonstrating superior stability and generalization.
- The results highlight that activation functions significantly influence learning dynamics, with SiLU/dSiLU providing smoother, self-regularizing nonlinearities that enhance value estimation and policy stability, especially in high-dimensional, long-horizon tasks.
- The experiments confirm that traditional on-policy methods, when combined with these novel activations, can match or exceed the performance of more complex off-policy algorithms like DQN, offering a simpler yet effective alternative for deep reinforcement learning.
Significance
This work challenges the prevailing reliance on experience replay and target networks in deep RL, proposing a streamlined on-policy framework with novel activation functions. The demonstrated performance gains in Tetris and Atari environments suggest a paradigm shift towards more efficient, stable, and interpretable RL algorithms. By emphasizing the role of activation functions, the study opens new avenues for neural network design, potentially reducing training complexity and resource requirements while maintaining high performance. The broader impact includes facilitating deployment in resource-constrained settings and advancing theoretical understanding of neural nonlinearities in RL.
Technical Contribution
The core technical innovation lies in the design of SiLU and dSiLU activation functions, which exhibit self-stabilizing properties and improve gradient flow. These functions enable networks to learn more robust representations without the need for complex regularization or architectural tricks. Coupled with on-policy TD(λ) and Sarsa(λ), the approach simplifies the RL pipeline, reduces hyperparameter sensitivity, and enhances value estimation accuracy. The integration of these components results in a more transparent and efficient learning process, with theoretical implications for the role of nonlinearities in reinforcement learning.
Novelty
This research is the first to systematically incorporate SiLU and dSiLU as activation functions within reinforcement learning frameworks, demonstrating their superiority over traditional units in high-dimensional, long-horizon tasks. Unlike prior work heavily reliant on experience replay and target networks, this approach leverages the intrinsic properties of these activations to stabilize learning and improve performance, representing a fundamental shift in neural network design for RL.
Limitations
- While effective in SZ-Tetris, 10×10 Tetris, and Atari, the approach's scalability to continuous control or more complex environments remains untested, potentially limiting its generality.
- Training deep networks still requires extensive hyperparameter tuning and computational resources, which may hinder real-time or embedded applications.
- The theoretical understanding of SiLU/dSiLU's properties in diverse RL settings is limited; further analysis is needed to establish formal guarantees and robustness.
Future Work
Future research will explore extending SiLU/dSiLU to continuous action spaces, integrating with policy gradient methods, and applying in robotics and autonomous systems. Additionally, theoretical investigations into activation function properties and their impact on convergence and generalization will be prioritized. Efforts to reduce computational costs and improve scalability will facilitate broader adoption in industry and real-world applications.
AI Executive Summary
Deep reinforcement learning has revolutionized AI, yet its reliance on complex architectures like experience replay and target networks limits efficiency and interpretability. This study introduces a novel approach by designing Sigmoid-weighted Linear Units (SiLU) and their derivatives (dSiLU) as activation functions, combined with on-policy TD(λ) and Sarsa(λ) algorithms. These innovations enable neural networks to learn effectively without experience replay, simplifying the RL pipeline. Extensive experiments on SZ-Tetris, 10×10 Tetris, and Atari 2600 demonstrate that networks utilizing SiLU/dSiLU outperform traditional units such as ReLU and sigmoid, achieving state-of-the-art scores. For instance, in Atari games, the SiLU-dSiLU deep network surpassed DQN by 232% on average, showcasing superior stability and generalization across diverse tasks. The key insight is that activation functions with self-stabilizing properties can significantly enhance value estimation and policy learning, especially in high-dimensional, long-horizon environments. This work challenges the conventional wisdom that complex off-policy methods are necessary for high performance, revealing that simpler on-policy strategies, when paired with well-designed nonlinearities, can achieve comparable or better results. The implications are profound: more efficient, transparent, and resource-friendly reinforcement learning algorithms are within reach, paving the way for broader deployment in real-world applications. Nonetheless, challenges remain in scaling to more complex environments and reducing training costs. Future directions include extending these methods to continuous control, integrating with policy gradient techniques, and deepening theoretical understanding of activation functions. Overall, this research marks a significant step towards more practical and robust reinforcement learning systems, with promising avenues for both academic inquiry and industrial innovation.
Deep Dive
Glossary
Sigmoid (S形函数)
一种平滑的非线性函数，将输入映射到0到1之间，用于激活神经元，帮助网络学习复杂模式。在论文中，用于构建SiLU激活函数。
作为SiLU的核心组成部分，调节网络的非线性表达。
TD(λ)
一种时间差分学习算法，结合eligibility traces实现对未来奖励的估计，提升学习效率和稳定性。
在本文中用于浅层和深层网络的on-policy训练。
Sarsa(λ)
一种基于策略的强化学习算法，结合eligibility traces进行动作值估计，适用于连续动作空间。
用于训练深度网络中的策略优化。
experience replay (经验回放)
存储过去经验以多次利用，提升样本效率，但增加架构复杂性。
本文摒弃该机制，采用on-policy策略。
DQN (Deep Q-Network)
结合深度卷积网络和Q-learning的强化学习算法，广泛应用于游戏和控制任务。
作为对比基准，本文旨在超越其性能。
Open Questions Unanswered questions from this research
- 1 激活函数SiLU和dSiLU在连续动作空间和复杂环境中的表现尚未充分验证，未来需系统性分析其泛化能力和理论基础。
- 2 如何进一步降低训练成本和提升模型在实际应用中的效率仍是待解难题。
Abstract
In recent years, neural networks have enjoyed a renaissance as function approximators in reinforcement learning. Two decades after Tesauro's TD-Gammon achieved near top-level human performance in backgammon, the deep reinforcement learning algorithm DQN achieved human-level performance in many Atari 2600 games. The purpose of this study is twofold. First, we propose two activation functions for neural network function approximation in reinforcement learning: the sigmoid-weighted linear unit (SiLU) and its derivative function (dSiLU). The activation of the SiLU is computed by the sigmoid function multiplied by its input. Second, we suggest that the more traditional approach of using on-policy learning with eligibility traces, instead of experience replay, and softmax action selection with simple annealing can be competitive with DQN, without the need for a separate target network. We validate our proposed approach by, first, achieving new state-of-the-art results in both stochastic SZ-Tetris and Tetris with a small 10$\times$10 board, using TD($λ$) learning and shallow dSiLU network agents, and, then, by outperforming DQN in the Atari 2600 domain by using a deep Sarsa($λ$) agent with SiLU and dSiLU hidden units.