Restructurable Activation Networks
Proposes RAN models with adjustable non-linearity, achieving hardware-efficient deep networks; RAN-e and RAN-i outperform baselines on ImageNet and hardware benchmarks.
Key Findings
Methodology
This paper introduces two models: RAN-explicit (RAN-e) employs a new search space and semi-automatic algorithm to replace inefficient blocks with hardware-friendly ones; RAN-implicit (RAN-i) leverages theoretical links between network topology and non-linear unit count to enable training-free model scaling. RAN-e uses AFRB structures to optimize hardware utilization, while RAN-i adjusts network depth and width based on topological metrics. Both are validated on ImageNet and various hardware platforms, demonstrating significant improvements in efficiency and speed.
Key Results
- RAN-e achieves comparable top-1 accuracy (~72.8%) to EfficientNet-Lite-B0 on ImageNet, with 1.5× FPS increase on Arm micro-NPU; RAN-i reduces MACs by 50% relative to ConvNexts while maintaining similar accuracy, with nearly 40% higher FPS on data center CPU; in object detection on COCO, RAN-i attains higher mAP and 33% FPS boost compared to ConvNext.
- Across multiple scales, RAN-e models outperform traditional architectures in hardware utilization and inference speed. RAN-i enables rapid, training-free model scaling based on topological metrics, showing superior performance across hardware types, validating the approach’s versatility.
Significance
This work pioneers the manipulation of activation non-linearity as a new pathway for hardware-aware neural network design. It addresses longstanding bottlenecks in hardware utilization and energy efficiency, offering a flexible framework adaptable to mobile, edge, and data center environments. The theoretical insights and practical results suggest a paradigm shift in how models are optimized for hardware, bridging the gap between architecture and hardware efficiency.
Technical Contribution
Key innovations include AFRB structures for explicit non-linearity control, a new search space and lightweight algorithm for hardware-aware design, and theoretical proof linking network topology to expressivity. Unlike traditional NAS, this work emphasizes non-linearity modulation for hardware efficiency, opening new avenues for model optimization and theoretical understanding of network capacity.
Novelty
This is the first systematic integration of activation function modulation into hardware-aware model design, using AFRB to enable explicit restructuring. The approach combines theoretical analysis of network topology with practical, training-free scaling, setting it apart from existing architecture search methods that focus solely on parameters or layer configurations. The dual framework of explicit and implicit restructuring offers a novel perspective on model optimization.
Limitations
- The current approach is validated mainly on specific hardware platforms; generalization to other hardware types requires further tuning. Model scaling based on topological metrics may not perform well under extreme constraints. AFRB structures may cause performance drops in certain tasks, necessitating further refinement.
Future Work
Future directions include integrating reinforcement learning or differentiable search algorithms for full automation, exploring multi-objective optimization balancing accuracy, latency, and energy, and extending the framework to other tasks like detection and segmentation. Further, improving theoretical understanding and hardware compatibility will enhance practical deployment.
AI Executive Summary
Deep neural networks have achieved remarkable success across numerous tasks, yet their deployment often faces hardware efficiency bottlenecks. Traditional models like ResNet and EfficientNet rely heavily on fixed activation functions such as ReLU, which, while computationally cheap, do not fully exploit the potential for hardware optimization. This paper introduces Restructurable Activation Networks (RANs), a novel framework that manipulates the non-linearity of activation functions to enhance hardware awareness and efficiency.
RAN-e employs a new search space and a semi-automatic algorithm to replace inefficient blocks with hardware-friendly structures called AFRBs. These blocks can explicitly restructure the network, reducing computational costs and improving hardware utilization without sacrificing accuracy. Experiments on ImageNet show that RAN-e achieves accuracy comparable to EfficientNet-Lite-B0 (~72.8%) while boosting inference speed by 1.5× on Arm micro-NPU.
Meanwhile, RAN-i leverages theoretical insights into network topology and the number of non-linear units to enable training-free model scaling. By adjusting network depth and width based on topological metrics, RAN-i reduces MACs by up to 50% with minimal accuracy loss, and achieves nearly 40% higher FPS on data center CPUs. In object detection tasks, RAN-i models outperform ConvNext in both mAP and FPS, demonstrating broad applicability.
These results highlight the potential of non-linearity modulation as a new paradigm for hardware-efficient neural network design. The approach offers a flexible, theoretically grounded method to optimize models for diverse hardware platforms, from mobile devices to data centers. Future work will focus on automating the search process further, exploring multi-objective optimization, and extending the framework to other vision tasks, promising a significant leap toward practical, efficient AI deployment.
Deep Analysis
Background
近年来，深度学习模型在准确率不断提升的同时，面临硬件资源有限的挑战。MobileNet、EfficientNet等架构通过参数剪枝和结构优化降低模型复杂度，但硬件利用率仍未充分解决。神经架构搜索（NAS）如DARTS、ProxylessNAS推动了自动化架构设计，但多集中在参数空间，忽视激活函数调控的重要性。硬件感知设计逐渐兴起，强调模型与硬件的协同优化，但缺少系统性调节激活非线性的方案。本文结合激活调控与硬件优化，提出新思路，填补该研究空白。
Core Problem
核心问题在于如何调节深度网络中的激活函数非线性程度，以提升硬件利用率和模型效率，同时保持或提升准确率。传统方法多依赖架构搜索或手工调优，成本高且缺乏理论基础。如何在无需训练的情况下，实现模型快速缩放以适应不同硬件资源，成为亟待解决的难题。
Innovation
提出AFRB结构，通过可调节参数实现激活函数的显式重构，支持模型块的动态调整。引入新搜索空间和轻量级算法，优化硬件利用率。理论上，证明网络拓扑与表达能力的关系，为模型缩放提供基础。区别于传统NAS，本研究强调非线性调控的硬件感知设计，兼顾模型性能与硬件效率，创新性强。
Methodology
- �� 设计AFRB结构，利用PReLU参数调节激活非线性程度，实现块的显式重构。• 构建包含多种AFRB（AFRB-1、AFRB-2、AFRB-3）的新搜索空间，支持硬件友好重组。• 使用半自动算法，优化α参数，选择低非线性或高非线性块。• 理论分析网络拓扑与非线性单元关系，指导模型缩放。• 在ImageNet上验证，通过调整模型结构适应不同硬件平台。• 结合硬件利用率指标筛选最优模型。
Experiments
在ImageNet上进行100和350轮训练，比较EfficientNet-Lite-B0、ConvNext、RAN-e、RAN-i在参数、MACs、准确率和FPS上的表现。硬件平台包括Arm微型NPU和数据中心CPU。通过消融实验验证AFRB贡献，分析模型缩放效果。利用性能估算器评估硬件利用率，确保实际表现。
Results
RAN-e在Arm微型NPU上实现72.8%准确率，FPS提升1.5倍，MACs与EfficientNet-Lite-B0相当。RAN-i在保持相似准确率的同时，MACs减半，FPS提升近40%。在数据中心CPU上，RAN-i FPS比ConvNext高近40%，目标检测中mAP更优，FPS提升33%。这些结果验证了调节激活非线性极大改善硬件效率，具有广泛应用潜力。
Applications
该技术适用于移动端、边缘设备和数据中心，优化模型在不同硬件上的性能表现。可用于自动化模型压缩、硬件感知架构设计及快速模型缩放，推动AI在实际场景中的部署。未来结合自动搜索算法，进一步提升模型效率和适应性。
Limitations & Outlook
目前方法主要在特定硬件平台验证，泛化到其他硬件仍需调优。模型缩放依赖拓扑指标，极端场景下效果有限。AFRB结构在某些任务中可能导致性能下降，需持续优化。
Plain Language Accessible to non-experts
想象你在厨房做饭，锅里有各种调料（激活函数），有些调料能让菜更香（模型更强），但用太多会浪费时间和能源（硬件资源）。这项研究就像发明了一种神奇的调料调节器，可以根据需要调整调料的用量，让菜既好吃又省时间。通过调节这个调料器，厨师可以快速改变菜的味道和做菜速度，不用换锅或买新材料。这就像模型中的激活函数一样，调节它们可以让模型在不同硬件上跑得更快、更省电，同时保持准确率。这项技术让厨房（模型设计）变得更智能、更高效，未来还能自动调节调料，做出最适合的菜。
ELI14 Explained like you're 14
你知道玩游戏时，有时候你想让角色跑得快点，但又不想牺牲太多技能？这就像科学家在设计神经网络时，也希望它既快又准。以前的方法就像用一样的跑鞋，无论跑多远都一样快，但其实不同的路面需要不同的鞋子。现在，这个研究发明了一种可以调节鞋底硬度的鞋子，你可以根据路面调节鞋子，让跑步既快又省力。它用一种特别的“调节器”控制激活函数的“硬度”，让模型在不同硬件上都能跑得又快又准。比如在手机上跑得快点，数据中心也能跑得更快。这就像你可以随时调节鞋子，让跑步变得更轻松。这项技术让神经网络变得更聪明、更灵活，就像你用遥控调节跑鞋一样酷！
Abstract
Is it possible to restructure the non-linear activation functions in a deep network to create hardware-efficient models? To address this question, we propose a new paradigm called Restructurable Activation Networks (RANs) that manipulate the amount of non-linearity in models to improve their hardware-awareness and efficiency. First, we propose RAN-explicit (RAN-e) -- a new hardware-aware search space and a semi-automatic search algorithm -- to replace inefficient blocks with hardware-aware blocks. Next, we propose a training-free model scaling method called RAN-implicit (RAN-i) where we theoretically prove the link between network topology and its expressivity in terms of number of non-linear units. We demonstrate that our networks achieve state-of-the-art results on ImageNet at different scales and for several types of hardware. For example, compared to EfficientNet-Lite-B0, RAN-e achieves a similar accuracy while improving Frames-Per-Second (FPS) by 1.5x on Arm micro-NPUs. On the other hand, RAN-i demonstrates up to 2x reduction in #MACs over ConvNexts with a similar or better accuracy. We also show that RAN-i achieves nearly 40% higher FPS than ConvNext on Arm-based datacenter CPUs. Finally, RAN-i based object detection networks achieve a similar or higher mAP and up to 33% higher FPS on datacenter CPUs compared to ConvNext based models. The code to train and evaluate RANs and the pretrained networks are available at https://github.com/ARM-software/ML-restructurable-activation-networks.