From Retrieved Context to Runtime Control: Adaptive Compression for Edge-based RAG
Proposes telemetry-guided adaptive compression for edge RAG, reducing GPU energy by up to 53.2% and SoC energy by 48.2%, with negligible quality loss.
Key Findings
Methodology
Using NVIDIA Jetson AGX Thor, experiments involved Llama and Qwen models, datasets Natural Questions and HotpotQA, and LLMLingua-2 compression. The study measured how compression rates affect latency, energy, and quality across models (1B-8B parameters), retrieval depths (k=5,10), and dynamic conditions. Telemetry data (temperature, delay, power) guided real-time compression adjustments. The approach identified the optimal tradeoff points, especially the two knees (energy and quality) in the compression-rate space, demonstrating the benefits of adaptive control over static strategies.
Key Results
- In models with 7B-8B parameters, generation accounts for about 90% of latency and 91% of GPU energy. Adaptive compression at rate 0.3 reduces GPU energy by up to 53.2% and SoC energy by 48.2%, with minimal impact on answer quality (F1 within ±0.05). Over-compression (rate 0.15) degrades responses significantly. Static compression strategies perform poorly across models; dynamic, telemetry-guided control maintains efficiency and quality.
- Across different models and retrieval depths, the optimal compression rate (~0.3) balances energy savings and response quality. The energy knee and quality knee points are consistent, with higher compression rates leading to increased energy consumption due to compressor overhead. The results validate that real-time telemetry can effectively inform compression decisions, maximizing net energy savings while preserving accuracy.
- The findings emphasize that edge RAG systems should incorporate live device telemetry to adaptively manage context compression. This approach enables significant energy reductions, especially in large models and long-context scenarios, without sacrificing response fidelity. It establishes a practical framework for intelligent resource management in constrained environments.
Significance
This work addresses a critical gap in edge AI deployment—balancing energy consumption with response quality in retrieval-augmented generation. Static compression strategies fail to adapt to workload variability and device state, leading to suboptimal energy use. By integrating telemetry data into dynamic control, the proposed method enables smarter, context-aware resource management, crucial for privacy-sensitive, latency-critical, and bandwidth-limited applications. It paves the way for more sustainable, autonomous edge AI systems capable of scaling with model size and workload complexity, thus advancing both research and industry practices in efficient edge inference.
Technical Contribution
The paper introduces a novel telemetry-informed adaptive compression framework that dynamically adjusts context pruning based on real-time device metrics. It combines a two-knee model (energy and quality) with a continuous compression rate parameter, enabling fine-grained control. The system leverages LLMLingua-2 for extractive compression, integrating it seamlessly into the RAG pipeline. This approach departs from static or offline tuning methods, offering a real-time, workload-aware solution that optimizes energy efficiency without compromising response accuracy. The experimental validation demonstrates substantial energy savings across multiple models and datasets, establishing a new paradigm for resource-aware edge AI.
Novelty
This is the first work to embed real-time telemetry data into the context compression decision process for edge RAG systems. Unlike prior static or offline approaches, this method adapts compression rates dynamically, guided by device state indicators such as temperature, latency, and power. The two-knee model provides a theoretical foundation for identifying optimal operating points, ensuring energy savings without degrading quality. This integration of hardware telemetry with adaptive NLP pipeline management represents a significant innovation in edge AI resource optimization.
Limitations
- The experiments are limited to FP16 models on NVIDIA Jetson hardware; applicability to larger models, different hardware, or quantized models remains to be validated. The extractive compression model (LLMLingua-2) may not capture all nuances, potentially affecting response quality in complex tasks. The reliance on accurate, real-time telemetry data introduces risks related to data noise, latency, or measurement errors, which could impair decision-making. Additionally, the current system focuses primarily on energy metrics, with less emphasis on latency or multi-task scenarios, which require further exploration.
Future Work
Future research will explore more efficient compression algorithms, such as streaming or hardware-aware methods, to further reduce overhead. Extending the framework to larger models and diverse hardware platforms, including quantized models, is essential. Integrating multiple RAG control knobs—retrieval depth, reranking, source selection—into a unified telemetry-guided policy will enhance overall system efficiency. Developing robust, multi-metric decision engines that consider latency, thermal constraints, and multi-user environments will further advance autonomous, energy-aware edge AI systems.
AI Executive Summary
The rapid growth of large language models (LLMs) has revolutionized natural language processing, yet deploying these models on edge devices remains challenging due to resource constraints. Traditional approaches focus on optimizing retrieval or model compression independently, often neglecting the interplay between context size, energy consumption, and response quality. This paper introduces a novel telemetry-guided adaptive compression framework for retrieval-augmented generation (RAG) systems operating at the edge. By leveraging real-time device telemetry—such as temperature, latency, and power—the system dynamically adjusts the amount of retrieved context passed to the generator, balancing energy efficiency with response fidelity.
Experimental validation on NVIDIA Jetson AGX Thor, involving models from 1B to 8B parameters and datasets like Natural Questions and HotpotQA, demonstrates that adaptive compression can reduce GPU energy consumption by up to 53.2% and SoC energy by 48.2%. The key insight is the identification of two critical operating points—the energy knee and the quality knee—beyond which further compression yields diminishing returns or degrades response quality. The results show that a moderate compression rate (~0.3) consistently offers optimal energy savings with negligible impact on accuracy, across various workloads.
This approach signifies a shift from static, offline tuning towards intelligent, real-time resource management. It enables edge AI systems to adapt to workload variability and hardware conditions, paving the way for more sustainable, autonomous, and scalable deployment of large language models in privacy-sensitive and latency-critical applications. The framework opens avenues for future research into multi-metric control, hardware-aware compression, and broader integration with other RAG pipeline parameters, ultimately fostering more efficient and intelligent edge AI ecosystems.
Deep Analysis
Background
Recent advances in large language models (LLMs) like GPT, Llama, and Qwen have significantly improved natural language understanding and generation capabilities. However, deploying these models on edge devices faces challenges related to computational cost, energy consumption, and latency. Retrieval-augmented generation (RAG) enhances model responses by conditioning on external knowledge retrieved at inference time, improving accuracy and freshness. Early efforts optimized retrieval speed and storage, but as models grew larger, the context length became a bottleneck, increasing memory and energy demands. Static context compression strategies were introduced to mitigate this, but they lacked adaptability to changing workloads and device states. With the rise of resource-constrained edge platforms like NVIDIA Jetson, there is a pressing need for dynamic, workload-aware resource management strategies that can optimize energy use without sacrificing response quality.
Core Problem
The core issue is balancing the retrieval context size with the resource limitations of edge devices. Longer contexts improve answer accuracy but significantly increase latency and energy consumption, especially in large models (7B-8B). Static compression methods fail to adapt to workload variability and device conditions, leading to suboptimal energy savings or degraded response quality. Moreover, the compression process itself consumes energy and computational resources, which can offset the benefits. Therefore, the challenge is to develop a real-time, adaptive control mechanism that dynamically manages context compression based on current device telemetry, workload demands, and model characteristics, ensuring optimal energy-performance tradeoffs.
Innovation
This work introduces a telemetry-guided adaptive compression framework that dynamically adjusts the context size in edge RAG systems. It employs a two-knee model—identifying the energy knee (where energy savings plateau) and the quality knee (where response quality drops)—to determine optimal compression points. The system leverages real-time telemetry data (temperature, latency, power) to inform decisions, enabling on-the-fly adjustments of the compression rate. Unlike prior static or offline tuning methods, this approach provides a continuous, workload-sensitive control mechanism. The integration of LLMLingua-2 for extractive compression, combined with a real-time decision engine, allows for significant energy savings (up to 53.2%) with minimal quality loss, demonstrating a novel synergy between hardware monitoring and NLP pipeline management.
Methodology
- �� Deploy RAG pipeline on NVIDIA Jetson AGX Thor, integrating Llama and Qwen models, with datasets Natural Questions and HotpotQA.
- �� Use LLMLingua-2 for extractive context compression, adjusting the compression rate from 0.15 to 1.0.
- �� Collect real-time telemetry data: temperature, latency, power, memory bandwidth.
- �� Model two critical points: energy knee (minimum energy point) and quality knee (acceptable quality threshold).
- �� Develop a control policy that, based on telemetry, decides whether to compress and at what rate.
- �� Evaluate performance through metrics: latency, energy (GPU and SoC), F1 score, recall.
- �� Compare static versus adaptive strategies across multiple workload scenarios to validate energy savings and quality retention.
Experiments
Experiments conducted on NVIDIA Jetson AGX Thor involved multiple models (1B, 3B, 8B parameters) and retrieval depths (k=5,10). Each configuration tested various compression rates, measuring latency, energy consumption, and answer quality (F1). The adaptive control policy used telemetry data to identify the optimal compression rate, especially focusing on the two knees. Results were statistically analyzed to determine the energy savings and quality tradeoffs. Additional ablation studies examined the impact of different telemetry signals and compression thresholds, ensuring robustness across workloads. The experimental setup aimed to simulate real-world edge deployment scenarios, with multiple runs to ensure statistical significance.
Results
The adaptive compression strategy achieved up to 53.2% GPU energy reduction and 48.2% SoC energy savings at a compression rate of 0.3, with negligible impact on F1 scores. The energy knee and quality knee points were consistent across models, confirming the validity of the two-knee model. Static compression strategies either wasted energy or degraded quality, highlighting the advantage of real-time telemetry-guided control. Larger models and longer retrieval contexts benefited most from adaptive compression, demonstrating scalability. The results validate that dynamic adjustment based on device telemetry significantly enhances energy efficiency without sacrificing response fidelity.
Applications
This framework is directly applicable to privacy-sensitive AI assistants, autonomous robots, augmented reality devices, and mobile applications where energy efficiency and latency are critical. It requires hardware capable of real-time telemetry collection and software for dynamic policy execution. The approach can be integrated into existing edge RAG pipelines, enabling smarter resource management, prolonging device battery life, and improving user experience in constrained environments. Long-term, it supports scalable deployment of large models on resource-limited devices, facilitating broader adoption of intelligent edge AI.
Limitations & Outlook
The current implementation relies on specific hardware (NVIDIA Jetson AGX Thor) and models (FP16 Llama, Qwen), requiring validation on other platforms and larger models. The extractive compression may lose critical information in complex tasks, affecting response quality. Telemetry data accuracy and latency impact decision effectiveness; noisy or delayed signals could impair control. The system primarily targets energy metrics, with less emphasis on latency or multi-task scenarios, which need further investigation. Future work should address these limitations to enhance robustness and generalizability.
Plain Language Accessible to non-experts
想象你在厨房做饭，食材准备、调料用量和烹饪时间都要根据厨房的实际情况灵活调整。有时候厨房很热，厨具也快用完，你就得节省用料，减少调料用量，但又不能让菜变得难吃。这个过程就像边缘设备在运行AI模型时，要根据设备的温度、能耗和反应速度，动态调整“调料”——也就是压缩上下文的程度。这样既能保证菜（回答）好吃（准确），又不会耗费太多电（能耗高）。如果太节省，菜就会变得平淡无味（回答质量差）；太放纵，又会浪费很多材料（能耗高）。因此，厨师（系统）需要根据厨房的实时情况，灵活调节用料，确保做出既好吃又节能的菜肴。这就是本文提出的遥测引导的自适应压缩策略的核心思想。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，你需要收集很多信息（像任务提示、地图、敌人位置），才能打赢。可是，信息太多了，游戏会变得很慢，也很费电。于是，你决定只看最重要的部分，把不太重要的内容省略掉，这样既能快点，又能省电。但是，如果省得太厉害，可能会遗漏关键线索，导致游戏失败。这个时候，你就需要一个聪明的助手，根据你玩游戏时的表现（比如反应速度、剩余电量、屏幕温度），告诉你什么时候可以省点信息，什么时候要多看点。这样，你既能快又能赢，还不费太多电。这就像论文里说的，用设备的实时数据帮忙调节信息的多少，让AI在边缘设备上既省电又能答得好。这个聪明的调节方法，让设备变得更聪明、更节能，也更适合在有限的空间和电池里工作。是不是很酷？
Abstract
Retrieval-augmented generation (RAG) improves language-model responses by grounding generation in external passages, which comes with overhead: retrieved context lengthens the prompt, increasing prefill work, KV-cache footprint, memory traffic, latency, and energy. Context compression offers a natural remedy by pruning retrieved text before generation. However, state-of-the-art context-compression methods are typically used with a fixed compression budget, or with the rate selected offline and then applied at inference time. This static view ignores both workload variation and the live state of the edge device. On an edge SoC, compression is not free: the compressor itself runs on the same SoC and consumes latency and energy that can offset any generation savings. This paper proposes a vision for telemetry-informed adaptive compression in edge RAG, grounded in experimental evidence. We characterize the compression tradeoff on the NVIDIA Jetson AGX Thor using Llama and Qwen generators, Natural Questions and HotpotQA datasets, and LLMLingua-2 compression. Our measurements show that generation dominates the RAG budget for larger models, reaching roughly 90% of per-query latency and 91% of GPU energy for 7B-8B generators. Exploring the impact of the compression rate reveals an adaptive operating region: mild compression can miss energy opportunities, and overly aggressive compression can hurt inference quality. Intermediate compression can reduce GPU energy by up to 53.2%, and SoC energy by up to 48.2%, with negligible quality loss. We argue for runtime policies that dynamically manage compression, guided by workload features and edge telemetry.