ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge--Cloud Speculative LLM Serving
ConfigSpec framework optimizes distributed inference configuration by analyzing edge devices and draft models.
Key Findings
Methodology
The ConfigSpec framework combines systematic device profiling, draft-target alignment measurement, and analytical performance modeling to evaluate configuration quality. By analyzing drafting throughput, acceptance rate, and device power draw, ConfigSpec maps these profiles to deployment-relevant metrics like goodput, verification cost efficiency, and energy efficiency.
Key Results
- On Jetson AGX Orin, the Llama-3.2-1B model achieves a goodput of approximately 6.5 tok/s, while the Qwen3-0.6B model reaches around 5.8 tok/s.
- Llama-3.1-8B achieves a cost efficiency of 913K tokens/$, while Qwen3-8B achieves 1224K tokens/$, improving by 19.4% and 24.9% respectively.
- In terms of energy efficiency, Llama-3.2-1B-Instruct Q4_K_M achieves 0.63 J/tok on Jetson and 1.28 J/tok on RPi 5.
Significance
The ConfigSpec framework significantly enhances the efficiency of large language model inference on edge devices by optimizing distributed inference configuration. This research addresses the challenge of selecting optimal configurations on heterogeneous hardware, advancing edge-cloud collaboration.
Technical Contribution
ConfigSpec provides a novel configuration evaluation method through systematic device profiling and model alignment measurement. Compared to existing methods, it not only improves inference efficiency but also offers new theoretical guarantees and engineering possibilities.
Novelty
ConfigSpec is the first to systematically combine device performance analysis and model alignment measurement to optimize distributed inference configuration. Compared to existing work, it offers a more comprehensive evaluation approach for configuration selection.
Limitations
- On resource-constrained platforms, larger draft models may lead to decreased energy efficiency.
- Verification latency is shared across all devices, potentially limiting additional drafting speed gains.
Future Work
Future research could explore more device types and model combinations to further optimize configuration selection and investigate how to reduce the impact of verification latency on inference efficiency.
AI Executive Summary
The ConfigSpec framework optimizes distributed inference configuration by analyzing edge devices and draft models. Existing solutions face challenges in selecting optimal configurations on heterogeneous hardware. ConfigSpec combines systematic device profiling, draft-target alignment measurement, and analytical performance modeling to evaluate configuration quality. Experimental results show that smaller draft models achieve the highest goodput on edge devices, while larger draft models perform better in cost efficiency. This research significantly enhances the efficiency of large language model inference on edge devices, advancing edge-cloud collaboration. However, verification latency is shared across all devices, potentially limiting additional drafting speed gains. Future research could explore more device types and model combinations to further optimize configuration selection.
Deep Analysis
Background
As the demand for large language models on edge devices increases, optimizing distributed inference configuration becomes crucial. Existing technologies face challenges in selecting optimal configurations on heterogeneous hardware. By analyzing device performance and model alignment, the ConfigSpec framework offers a novel solution.
Core Problem
Selecting optimal configurations to optimize distributed inference efficiency on heterogeneous hardware is a complex issue. Larger draft models may improve acceptance rates but consume more computational resources, affecting energy efficiency.
Innovation
The ConfigSpec framework combines device performance analysis and model alignment measurement to provide a novel configuration evaluation method. It analyzes drafting throughput, acceptance rate, and device power draw, mapping these profiles to deployment-relevant metrics.
Methodology
- �� Analyze device performance, measuring drafting throughput and acceptance rate.
- �� Measure device power draw, evaluating energy efficiency.
- �� Use analytical performance modeling to evaluate configuration quality, optimizing goodput and cost efficiency.
Experiments
Experiments were conducted on three edge platforms: Raspberry Pi 4B, Raspberry Pi 5, and NVIDIA Jetson AGX Orin. The Databricks Dolly 15K dataset was used for testing, evaluating the configuration effects of different draft models and target models.
Results
Experimental results show that smaller draft models achieve the highest goodput on edge devices, while larger draft models perform better in cost efficiency. Energy efficiency analysis shows that smaller draft models perform best in terms of energy efficiency.
Applications
The ConfigSpec framework can be used to optimize large language model inference configuration on edge devices, improving inference efficiency and cost-effectiveness. It is suitable for industries requiring efficient inference, such as healthcare and industrial IoT.
Limitations & Outlook
Verification latency is shared across all devices, potentially limiting additional drafting speed gains. Larger draft models may lead to decreased energy efficiency, and future research could explore more device types and model combinations.
Plain Language Accessible to non-experts
Imagine a factory where workers need to quickly produce products and check their quality. The factory has two departments: one for rapid production and another for quality inspection. ConfigSpec is like the factory manager, responsible for selecting the most suitable production line and inspection process to ensure product quality and production efficiency. By analyzing workers' speed and inspection efficiency, ConfigSpec helps the factory optimize its production process.
ELI14 Explained like you're 14
Imagine you're playing a game where you need to choose different characters to complete tasks. Some characters run fast but have low attack power, while others have high attack power but are slow. ConfigSpec is like a strategy guide in the game, helping you choose the best character combination to complete tasks. By analyzing characters' speed and attack power, ConfigSpec helps you win the game.
Glossary
Speculative Decoding
A decoding paradigm where a lightweight draft model proposes multiple candidate tokens ahead of time, verified by a larger target model.
Used in the paper to optimize distributed inference configuration.
Goodput
Refers to the throughput of verified tokens, defined as the verified token throughput under the speculative decoding round model.
Used to evaluate configuration quality.
Verification Cost Efficiency
The number of accepted tokens per dollar, depending on acceptance rate, speculative length, and verifier price.
Used to evaluate the cost-effectiveness of configurations.
Energy Efficiency
Energy consumption per verified token, calculated as the average power draw during drafting.
Used to evaluate the energy efficiency of configurations.
Draft Model
A lightweight model used to propose candidate tokens ahead of time and align with the target model.
Used in the paper for speculative decoding.
Open Questions Unanswered questions from this research
- 1 How to further reduce the impact of verification latency on inference efficiency remains an open question.
- 2 Exploring more device types and model combinations for optimizing configuration selection requires further research.
Applications
Immediate Applications
Medical Diagnosis
By optimizing inference configuration, improve the efficiency and accuracy of medical diagnosis. Suitable for scenarios requiring rapid response.
Long-term Vision
Smart Cities
By optimizing inference configuration on edge devices, enhance the efficiency and responsiveness of smart cities. Requires overcoming device heterogeneity and data privacy issues.
Abstract
Speculative decoding enables collaborative Large Language Model (LLM) inference across cloud and edge by separating lightweight token drafting from heavyweight verification. While prior systems show performance and cost benefits, practical deployment requires navigating a large configuration space spanning draft model variants, quantisation levels, speculative lengths, and heterogeneous edge devices. This paper presents ConfigSpec, a configurationselection framework for distributed speculative LLM serving. ConfigSpec profiles edge devices and draft-target alignment, and models drafting throughput, acceptance rate, and power to evaluate goodput, verification cost efficiency, and energy efficiency across the joint configuration space. Our analysis across three edge platforms and two LLM families reveals structurally conflicting optima. Firstly, goodput is maximised by the smallest, fastest draft model at device-dependent speculative lengths (K*=2-10). Secondly, both cost and energy efficiency converge to K=2 due to a dominant bonus-token effect-with cost favouring the largest drafter for its high acceptance rate and energy favouring the smallest for its low power draw. These conflicts confirm that no single fixed configuration can simultaneously optimise all objectives, underscoring the need for profiling-based configuration selection in disaggregated edge-cloud LLM inference.