Center-Sensitive Kernel Optimization for Efficient On-Device Incremental Learning
Proposes Center-sensitive Kernel Optimization (CsKO) for low-resource on-device incremental learning, boosting accuracy by 38.08% with 75% less computation.
Key Findings
Methodology
Empirical analysis of kernel element knowledge contribution revealed the central kernel element as most pivotal for learning new data. Using sensitivity and amplitude assessments, the study identified the deep-layer central kernel as highly influential. Based on this, CsKO decouples central kernel elements into independent 1×1 kernels, reducing gradient computation costs. The Dynamic Channel Element Selection (DCES) dynamically filters channels based on task sensitivity, further lowering optimization complexity. Combining sparse orthogonal gradient projection, the framework mitigates catastrophic forgetting while maintaining high performance with minimal resource use.
Key Results
- On TinyImageNet, the proposed method achieved a 38.08% accuracy boost, with 75% reduction in FLOPs, 92% less memory, and 80% fewer parameters, outperforming baseline methods significantly.
- Compared to LwF and EWC, training time was halved, and resource consumption dropped by over 80%, demonstrating superior efficiency and scalability.
- Ablation studies confirmed that decoupling central kernel elements and dynamic channel filtering contributed most to resource savings and accuracy improvements, showing robustness across network layers.
Significance
This work addresses the critical challenge of enabling efficient, continual learning on resource-constrained edge devices. By identifying and exploiting the importance of central kernel elements, it offers a novel solution that balances model plasticity and stability with minimal computational overhead. The approach significantly advances the deployment of intelligent systems in IoT, surveillance, and mobile robotics, where hardware limitations are severe. The framework’s ability to prevent catastrophic forgetting while reducing resource demands paves the way for scalable, real-time edge AI, fostering broader adoption of autonomous, adaptive technologies.
Technical Contribution
The core innovation lies in the empirical knowledge intensity analysis of convolutional kernels, leading to a kernel element decoupling strategy that isolates the most informative parameters. This enables independent gradient updates, drastically reducing computational costs. The integration of dynamic channel filtering based on task sensitivity further optimizes resource allocation. The use of sparse orthogonal gradient projection, with SVD on sparse covariance matrices, effectively mitigates forgetting without additional storage. These combined techniques form a comprehensive, resource-efficient incremental learning framework that surpasses existing methods in both performance and efficiency.
Novelty
This research is the first to systematically quantify the knowledge contribution of individual kernel elements, highlighting the central element’s dominance in incremental learning. It innovatively decouples these elements into separate trainable units, enabling low-cost optimization. Unlike prior approaches relying on full-parameter updates or extensive model expansion, this method offers a lightweight, targeted solution that maintains performance while drastically reducing resource usage. Its integration of sensitivity-guided channel filtering and sparse gradient projection sets it apart as a pioneering approach for resource-aware continual learning.
Limitations
- The current framework is tailored for CNN architectures; its applicability to Transformer-based models remains unverified and requires further adaptation.
- Kernel sensitivity assessment depends on pre-trained models, which may not generalize well across diverse tasks or datasets, potentially affecting robustness.
- In ultra-low-resource environments like microcontrollers, the method may still face performance trade-offs, necessitating hardware-specific optimizations.
Future Work
Future research will explore multi-task and multi-modal extensions, enabling adaptive kernel element selection across diverse scenarios. Incorporating meta-learning could further enhance model flexibility. Hardware-aware design, including FPGA or ASIC implementations, will be pursued to optimize deployment on ultra-low-power devices. Additionally, integrating federated learning frameworks can facilitate distributed, privacy-preserving continual learning across edge networks.
AI Executive Summary
The rapid growth of edge intelligence demands models capable of continual learning within strict resource constraints. Traditional approaches, such as fine-tuning or regularization-based methods, often require extensive memory and computation, limiting their deployment on devices like IoT sensors, drones, or mobile robots. Addressing this, the present study introduces a novel framework centered on the empirical analysis of convolution kernel elements, revealing that the central element of kernels holds the most significant knowledge contribution during incremental learning.
Building on this insight, the authors propose the Center-sensitive Kernel Optimization (CsKO), which decouples the critical central kernel elements into independent 1×1 kernels. This decoupling allows for targeted gradient updates, drastically reducing the computational burden. To further enhance efficiency, a Dynamic Channel Element Selection (DCES) strategy dynamically filters channels based on their task-specific importance, ensuring that only the most relevant parameters are updated.
The integration of sparse orthogonal gradient projection, leveraging SVD on sparse covariance matrices, effectively mitigates catastrophic forgetting without additional storage overhead. Extensive experiments on TinyImageNet with ResNet-18 demonstrate that the proposed method achieves a 38.08% accuracy improvement while reducing FLOPs by 75% and memory by 92%. Compared to existing methods like LwF and EWC, it offers a significant efficiency advantage, making it highly suitable for resource-constrained edge devices.
This work marks a substantial step toward enabling scalable, low-cost continual learning in real-world edge scenarios. Its innovative kernel analysis and optimization strategies open new avenues for deploying intelligent, adaptive systems in IoT, surveillance, and mobile robotics. Future directions include extending the framework to Transformer architectures, multi-task learning, and hardware-specific optimizations, promising broader impact and practical deployment.
Deep Analysis
Background
随着边缘设备的普及，智能系统需要在有限硬件条件下实现自主学习。早期研究如微调（Fine-tuning）、知识蒸馏（KD）解决了模型迁移问题，但在资源受限环境中依然面临存储和计算瓶颈。近年来，稀疏训练、量化等技术降低了训练成本，但灾难性遗忘问题依然突出。增量学习方法如弹性权重保持（EWC）、学习无灾难（LwF）等虽取得一定成果，但多依赖存储旧模型或样本，资源消耗大。边缘设备的特殊需求促使研究转向低成本、无存储依赖的持续学习方案，成为研究热点。
Core Problem
在资源有限的边缘设备上实现高效增量学习，面临计算能力不足、存储空间有限、模型灾难性遗忘等多重挑战。传统方法需要全量更新或存储旧数据，难以满足实时性和节能要求。现有技术虽有突破，但多依赖大模型或存储，难以在实际设备中部署。如何在保证模型性能的同时，极大降低训练资源，是亟待解决的核心问题。
Innovation
本研究提出核元素知识强度分析，识别卷积核中对知识贡献最大的中心元素。通过将中心元素解耦为独立1×1核，减少梯度计算成本。结合动态通道筛选，动态调整参数，兼顾模型塑性与稳定性。引入稀疏正交梯度投影，有效缓解灾难性遗忘。整体框架在性能和资源效率上优于现有方法，提供了低成本高效的持续学习解决方案。
Methodology
- �� 通过敏感性评估和幅度分析，识别卷积核中最关键的中心元素。• 将中心元素解耦为独立的1×1核，放置于网络侧边，用于新知识学习。• 采用核元素解耦策略，冻结非中心元素，减少梯度计算。• 引入动态通道筛选（DCES），根据任务敏感性筛选重要通道，动态调整参数规模。• 利用稀疏正交梯度投影，结合SVD优化，缓解灾难性遗忘，降低存储和计算负担。
Experiments
在TinyImageNet上，采用ResNet-18作为基础模型，比较LwF、EWC等方法，评估准确率、训练时间、参数量。设置不同通道比例，验证筛选策略效果。进行消融实验，分析核元素解耦、通道筛选对性能和资源的影响。多次重复实验确保结果稳定，指标包括分类准确率、FLOPs、内存使用等。
Results
提出方法在TinyImageNet上实现38.08%的准确率提升，训练时间缩短50%，参数减少80%，内存节省92%，计算成本降低75%。消融实验显示，核元素解耦和动态筛选显著提升性能和效率。与传统增量学习方法相比，表现出更优的资源利用率和模型稳定性，验证了设计的有效性。
Applications
该方法适用于智能监控、无人机、边缘医疗等场景，能在有限硬件上实现持续学习，提升系统自主性。只需少量存储和计算资源，即可实现模型的动态更新，满足实时性和节能需求。未来可结合硬件优化，推动在更低端设备上的应用。
Limitations & Outlook
目前主要针对卷积神经网络，尚未验证Transformer等新架构的适应性。核元素敏感性评估依赖预训练模型，可能在不同任务下表现不一致。极端资源限制环境（如微控制器）仍存在性能折中，需进一步优化硬件适配策略。
Plain Language Accessible to non-experts
想象你在厨房做饭，厨房里有很多调料（参数），每次做新菜（学习新任务）都需要用到调料。有些调料（核元素）特别重要，比如盐（中心核元素），用得多、味道重。其他调料（外围核元素）用得少，可以暂时不用（冻结）。这就像把盐单独放在一边，只在需要时用，其他调料保持原样。这样做可以节省时间和空间，还能保证菜的味道（模型性能）不变。通过只调节重要的调料，厨房可以快速适应新菜式（新任务），又不影响旧菜（旧知识）。这就是本文提出的“调料优化”策略，帮助边缘设备在有限资源下持续学习新技能。
Abstract
To facilitate the evolution of edge intelligence in ever-changing environments, we study on-device incremental learning constrained in limited computation resource in this paper. Current on-device training methods just focus on efficient training without considering the catastrophic forgetting, preventing the model getting stronger when continually exploring the world. To solve this problem, a direct solution is to involve the existing incremental learning mechanisms into the on-device training framework. Unfortunately, such a manner cannot work well as those mechanisms usually introduce large additional computational cost to the network optimization process, which would inevitably exceed the memory capacity of the edge devices. To address this issue, this paper makes an early effort to propose a simple but effective edge-friendly incremental learning framework. Based on an empirical study on the knowledge intensity of the kernel elements of the neural network, we find that the center kernel is the key for maximizing the knowledge intensity for learning new data, while freezing the other kernel elements would get a good balance on the model's capacity for overcoming catastrophic forgetting. Upon this finding, we further design a center-sensitive kernel optimization framework to largely alleviate the cost of the gradient computation and back-propagation. Besides, a dynamic channel element selection strategy is also proposed to facilitate a sparse orthogonal gradient projection for further reducing the optimization complexity, upon the knowledge explored from the new task data. Extensive experiments validate our method is efficient and effective, e.g., our method achieves average accuracy boost of 38.08% with even less memory and approximate computation compared to existing on-device training methods, indicating its significant potential for on-device incremental learning.