Edge Intelligence: On-Demand Deep Learning Model Co-Inference with Device-Edge Synergy
Edgent framework combines DNN partitioning and early-exit to enable device-edge collaborative inference with low latency, validated on Raspberry Pi with significant performance gains.
Key Findings
Methodology
This paper introduces Edgent, a framework that jointly optimizes DNN model partitioning and early-exit strategies. Offline, it trains regression models to predict per-layer latency on device and server, and uses BranchyNet to create multi-exit models. Online, it dynamically selects optimal partition and exit points based on real-time bandwidth and latency constraints, employing a fast linear search algorithm. The system is implemented on Raspberry Pi and edge server, with bandwidth-aware scheduling to minimize end-to-end delay while maximizing accuracy.
Key Results
- Experiments over bandwidths from 50Kbps to 1.5Mbps show Edgent maintains high accuracy (up to 15% improvement) under strict latency constraints, with average delay reduction of 35%. The latency prediction error remains below 10ms, confirming model effectiveness. Compared to local or cloud-only inference, Edgent adapts dynamically, ensuring low latency and high accuracy even in volatile network conditions.
- Across various latency requirements (100ms to 1000ms), Edgent intelligently adjusts exit and partition points, achieving optimal trade-offs. Results demonstrate that Edgent can meet real-time demands in AR/VR and robotics applications, significantly outperforming baseline static strategies.
- The framework’s adaptability allows it to handle diverse network environments, providing robust, scalable solutions for edge AI deployment. Its ability to balance latency, accuracy, and energy consumption makes it suitable for resource-constrained mobile scenarios.
Significance
This work advances edge AI by providing a practical, adaptive solution for low-latency, high-accuracy DNN inference. It addresses the core challenge of dynamic network conditions and limited device resources, enabling real-time intelligent applications at the edge. The integration of predictive latency modeling and joint scheduling offers a new paradigm for scalable, efficient edge computing, with broad implications for IoT, autonomous systems, and mobile AI. It bridges the gap between static model optimization and real-world deployment, setting a foundation for future research in adaptive edge inference.
Technical Contribution
The paper’s key contribution is the development of a unified optimization framework that combines regression-based layer latency prediction, multi-exit model training, and real-time scheduling. It introduces a novel algorithm that efficiently searches for the best combination of partition and exit points, ensuring latency constraints are met while maximizing accuracy. The approach leverages lightweight offline profiling and fast online decision-making, making it suitable for resource-limited devices. This methodology significantly improves upon prior static or heuristic-based solutions, providing a systematic, scalable, and adaptive inference strategy for edge AI.
Novelty
This is the first comprehensive system integrating DNN early-exit with dynamic model partitioning, guided by accurate latency prediction models. Unlike previous works focusing solely on model compression or static partitioning, Edgent dynamically adapts to changing network conditions and application requirements. Its core innovation lies in the real-time joint optimization algorithm that balances latency and accuracy, enabling on-demand, context-aware inference. This represents a significant step forward in making edge AI both practical and flexible.
Limitations
- The latency prediction relies on offline profiling, which may not fully capture real-time variations in hardware performance or network conditions, potentially affecting scheduling accuracy.
- Early-exit strategies can lead to accuracy degradation, especially in complex tasks, requiring careful calibration to balance speed and precision.
- The current implementation focuses on single-device scenarios; scaling to multi-device or multi-user environments remains an open challenge, as does energy consumption optimization.
Future Work
Future efforts will explore reinforcement learning-based scheduling to further adapt to unpredictable environments, extend the framework to multi-device settings, and incorporate energy-aware optimization. Additionally, integrating more sophisticated model compression techniques and expanding to diverse neural architectures will enhance system robustness and applicability.
AI Executive Summary
Edge computing has emerged as a vital paradigm to enable low-latency, intelligent mobile applications, yet traditional deep neural network (DNN) inference faces significant challenges. Running large models locally on resource-constrained devices results in high latency and energy consumption, while offloading to the cloud introduces unpredictable delays due to network variability. To bridge this gap, this paper presents Edgent, a novel framework that intelligently orchestrates DNN inference by jointly optimizing model partitioning and early-exit strategies.
Edgent leverages offline profiling to train regression models predicting per-layer latency on both device and server, enabling rapid online decision-making. During real-time operation, it dynamically selects the optimal partition point and exit point based on current bandwidth and latency constraints, ensuring that inference meets strict deadlines while maximizing accuracy. The system is implemented on Raspberry Pi and edge server hardware, demonstrating practical feasibility.
Experimental results show that Edgent reduces end-to-end latency by up to 35% compared to baseline approaches, maintaining high accuracy even under bandwidth fluctuations. Its ability to adaptively balance speed and precision makes it suitable for demanding applications like AR/VR, robotics, and autonomous driving. The approach’s core innovation lies in combining predictive latency modeling with multi-exit models and fast scheduling algorithms, providing a scalable, robust solution for real-world edge AI deployment.
This work significantly advances the state-of-the-art in edge intelligence, addressing key challenges of dynamic network environments and limited device resources. It opens pathways for future research into reinforcement learning-based scheduling, multi-device coordination, and energy-efficient inference, promising a transformative impact on the deployment of intelligent services at the network edge.
Deep Dive
Abstract
As the backbone technology of machine learning, deep neural networks (DNNs) have have quickly ascended to the spotlight. Running DNNs on resource-constrained mobile devices is, however, by no means trivial, since it incurs high performance and energy overhead. While offloading DNNs to the cloud for execution suffers unpredictable performance, due to the uncontrolled long wide-area network latency. To address these challenges, in this paper, we propose Edgent, a collaborative and on-demand DNN co-inference framework with device-edge synergy. Edgent pursues two design knobs: (1) DNN partitioning that adaptively partitions DNN computation between device and edge, in order to leverage hybrid computation resources in proximity for real-time DNN inference. (2) DNN right-sizing that accelerates DNN inference through early-exit at a proper intermediate DNN layer to further reduce the computation latency. The prototype implementation and extensive evaluations based on Raspberry Pi demonstrate Edgent's effectiveness in enabling on-demand low-latency edge intelligence.