CarbonCall: Sustainability-Aware Function Calling for Large Language Models on Edge Devices
CarbonCall dynamically selects tools and adjusts model precision based on real-time carbon intensity, reducing emissions by up to 52% while maintaining performance.
Key Findings
Methodology
CarbonCall integrates a vector-based tool retrieval system with a cross-encoder re-ranking module, enabling efficient and accurate API selection. It employs a real-time carbon intensity forecast to dynamically modulate device power limits via predefined hardware modes. Additionally, it adopts a multi-quantization approach, switching between full-precision and low-bit models (Q8, Q4K M) based on energy constraints. The system continuously monitors throughput, adjusting hardware configurations and model precision to balance responsiveness and energy savings. This comprehensive framework ensures sustainable, high-performance function calling on edge devices, validated through experiments on NVIDIA Jetson AGX Orin with diverse benchmarks.
Key Results
- Across multiple weeks, CarbonCall achieved up to 52% reduction in carbon emissions, 30% decrease in power consumption, and 30% faster execution times, outperforming baseline methods like Gorilla and LiS. It maintained high tokens/sec throughput with minimal performance degradation, demonstrating effective adaptation to fluctuating carbon intensity levels. The tool selection accuracy was improved via cross-encoder re-ranking, reducing miscalls and unnecessary energy expenditure.
- In scenarios with high carbon intensity variability, the system dynamically adjusted hardware modes and model quantization levels, ensuring optimal trade-offs. Results showed consistent performance across different models (Hermes2, Llama3.1, Qwen2) and datasets, confirming robustness. The experiments validated that environmental-aware scheduling significantly outperforms static or offline strategies in both emissions and efficiency.
- Furthermore, the adaptive switching between quantized models based on throughput thresholds preserved response times, while reducing energy use. The integrated approach effectively mitigated the impact of environmental fluctuations, establishing a new standard for sustainable edge AI deployments.
Significance
This work addresses a critical challenge in deploying large language models at the edge—balancing high responsiveness with low environmental impact. By integrating real-time carbon intensity data into the scheduling and model selection process, CarbonCall offers a scalable, practical solution for sustainable AI. It paves the way for environmentally conscious edge computing, aligning AI development with global climate goals. The framework's adaptability to diverse hardware and workload scenarios makes it highly relevant for industry adoption, especially in resource-constrained settings aiming for carbon neutrality.
Technical Contribution
The paper introduces a novel multi-component framework combining vector similarity search, cross-encoder re-ranking, real-time carbon-aware hardware scheduling, and multi-quantization model switching. It advances the state-of-the-art by embedding environmental feedback directly into the inference pipeline, enabling dynamic, context-aware optimization. The approach offers theoretical guarantees on energy savings and responsiveness, supported by extensive empirical validation. This integrated design opens new avenues for research in sustainable AI and hardware-aware model deployment.
Novelty
This is the first work to embed real-time carbon intensity forecasting into the dynamic scheduling and model selection process for edge LLMs. Unlike prior static or offline optimization methods, CarbonCall achieves environment-aware adaptation through a seamless integration of tool retrieval, hardware configuration, and model quantization. Its multi-layered approach to balancing energy efficiency and performance represents a significant innovation in sustainable AI deployment.
Limitations
- The system's effectiveness relies on the accuracy of carbon intensity forecasts; inaccuracies can lead to suboptimal scheduling and碳排放控制。
- Quantization introduces some loss in model accuracy, especially in complex reasoning tasks, which may affect API call correctness.
- Extreme low-power modes can cause response delays and reduce model fidelity, requiring further refinement of transition strategies.
Future Work
Future research will focus on integrating reinforcement learning to optimize scheduling policies further, incorporating multi-device coordination for large-scale deployments, and enhancing prediction models for more precise environmental adaptation. Exploring adaptive granularity in model quantization and hardware configurations will also be key to achieving even finer energy-performance trade-offs, facilitating broader adoption in diverse edge scenarios.
AI Executive Summary
The rapid proliferation of large language models (LLMs) in edge AI applications has brought unprecedented capabilities but also significant energy consumption and environmental concerns. Traditional optimization strategies primarily target performance, often neglecting the carbon footprint associated with deploying these models on resource-constrained devices. As edge devices increasingly operate in regions with varying energy sources, the need for environmentally adaptive AI systems becomes urgent.
This paper introduces CarbonCall, a novel framework designed to enable sustainable, high-efficiency function calling for LLMs on edge devices. The core innovation lies in integrating real-time carbon intensity forecasts with dynamic hardware and model management. By employing a vector-based tool retrieval system combined with a cross-encoder re-ranking mechanism, CarbonCall ensures accurate and efficient API selection. Simultaneously, it adapts device power limits based on predicted environmental impact, utilizing predefined hardware modes that balance energy consumption and responsiveness.
A key technical feature is the use of multiple quantized models, switching between full-precision and low-bit versions (Q8, Q4K M) depending on energy constraints. This approach maintains high throughput and low latency even under strict power caps. The system continuously monitors throughput and environmental data, adjusting hardware configurations and model precision in real time. Extensive experiments on NVIDIA Jetson AGX Orin across diverse benchmarks demonstrate that CarbonCall reduces carbon emissions by up to 52%, cuts power consumption by 30%, and shortens execution times by 30%, outperforming existing methods.
Beyond performance metrics, this work signifies a paradigm shift toward environmentally conscious edge AI. By embedding sustainability into the core inference pipeline, CarbonCall addresses the pressing challenge of deploying large models responsibly. Its adaptive, environment-aware scheduling framework offers a scalable solution for future AI systems committed to reducing their carbon footprint while maintaining high responsiveness. Future directions include integrating reinforcement learning for even smarter scheduling, expanding multi-device coordination, and refining environmental prediction models to further enhance sustainability in AI deployment.
Deep Dive
Plain Language Accessible to non-experts
想象你在一个工厂里工作，工厂每天都要生产不同的产品。为了节省能源和减少污染，工厂会根据当天的能源供应情况调整生产速度和使用的机器。有时候能源很充裕，工厂会用最快的机器，生产效率高；有时候能源紧张，工厂就会用节能的机器，生产变慢，但更环保。这个工厂还会根据天气和能源供应的变化，自动调整机器的工作状态，确保既不浪费能源，又能按时完成任务。CarbonCall就像这个智能工厂，能根据环境变化智能调节机器的工作方式，既保证效率，又保护环境。
ELI14 Explained like you're 14
你知道吗，就像我们用手机玩游戏，有时候电池快没电了，我们会把游戏调低画质，省点电，虽然画面变差了，但能用更久。类似的，CarbonCall会根据电力的环保程度，自动调节电脑的工作速度和用的模型的精度。当天如果用的电来自很多清洁能源，它会让电脑跑得快点，效果好一些；如果电来自煤炭等污染大源，它就会让电脑用低点的模型，少点能耗，减少污染。这样一来，不管外面能源怎么变，电脑都能既快又环保，就像我们聪明地调节手机一样！
Abstract
Large Language Models (LLMs) enable real-time function calling in edge AI systems but introduce significant computational overhead, leading to high power consumption and carbon emissions. Existing methods optimize for performance while neglecting sustainability, making them inefficient for energy-constrained environments. We introduce CarbonCall, a sustainability-aware function-calling framework that integrates dynamic tool selection, carbon-aware execution, and quantized LLM adaptation. CarbonCall adjusts power thresholds based on real-time carbon intensity forecasts and switches between model variants to sustain high tokens-per-second throughput under power constraints. Experiments on an NVIDIA Jetson AGX Orin show that CarbonCall reduces carbon emissions by up to 52%, power consumption by 30%, and execution time by 30%, while maintaining high efficiency.