Efficient On-Device Agents via Adaptive Context Management
Proposes an on-device AI framework with adaptive memory and tool management, reducing context size by over 6x while maintaining performance.
Key Findings
Methodology
This work introduces a dual-adapter system leveraging LoRA modules to compress conversational history into a structured Context State Object (CSO). It employs a minimalistic serialization format for tool schemas and a JIT mechanism that loads full tool definitions only upon invocation. The model, based on a 3B parameter Sparse Language Model (SLM), is fine-tuned with simulated multi-turn dialogues and evaluated on complex tasks. The architecture involves: • Using LoRA adapters (Executor and Mem) for history compression and state tracking; • Designing a lightweight, token-efficient tool schema format; • Implementing a JIT loading strategy for tools. The system achieves significant context compression while preserving task accuracy.
Key Results
- On complex multi-turn tasks, the model attains a tool call F1 score of 0.99, outperforming baseline models with 0.41. The initial context size drops from 3200 tokens to 400 tokens, an 8x reduction. Context growth rate over long dialogues decreases by 10-25 times, enabling persistent interactions.
- Tool call precision reaches 97%, recall 95%, with an overall F1 above 0.98, demonstrating effective tool management. The structured CSO and JIT loading contribute to maintaining high performance under strict memory constraints.
- Ablation studies confirm that the combined approach of structured memory and on-demand schema loading yields the best trade-off between efficiency and accuracy, supporting scalable on-device AI deployment.
Significance
This research addresses the critical challenge of limited memory in edge devices, enabling complex, multi-turn interactions without sacrificing privacy or responsiveness. The proposed framework offers a scalable solution for deploying capable AI assistants on smartphones and IoT devices, pushing forward the frontier of personalized, low-latency AI. It bridges the gap between large models’ capabilities and edge hardware limitations, fostering broader adoption of intelligent on-device systems.
Technical Contribution
The main innovations include: • A dual-adapter architecture with LoRA modules for semantic history compression; • A token-efficient, minimal tool schema serialization format; • A JIT mechanism that loads full tool schemas only when necessary. These components collectively enable high-performance multi-turn dialogue with constrained memory, offering a new paradigm for edge AI that balances efficiency, scalability, and task fidelity.
Novelty
This work uniquely combines structured semantic memory with selective, on-demand tool schema loading tailored for edge devices. Unlike prior long-context models (e.g., Longformer, RETRO), which require extensive hardware resources, this approach achieves comparable or better performance within strict memory budgets. It is the first to systematically address the integration of adaptive memory and tool management for persistent, multi-tool AI agents on resource-limited hardware.
Limitations
- Despite significant compression, the model may still struggle with extremely complex tasks involving many tools or very long dialogues, risking information loss.
- JIT loading introduces latency, especially when frequent tool schema updates are needed, potentially impacting real-time responsiveness.
- The training relies on simulated data, which may not cover all real-world variability, necessitating further fine-tuning for diverse scenarios.
Future Work
Future directions include integrating multimodal inputs, developing dynamic schema update mechanisms, and optimizing hardware-aware inference. Expanding training datasets with real user interactions and exploring adaptive context compression strategies will further enhance robustness and generalization, pushing edge AI towards more autonomous and versatile applications.
AI Executive Summary
In recent years, the deployment of AI models directly on edge devices such as smartphones and IoT gadgets has gained momentum, driven by privacy concerns and the need for low-latency responses. However, these devices face significant limitations in memory and computational capacity, constraining the ability of large language models (LLMs) to support rich, multi-turn interactions with complex tools. Traditional approaches, relying on extensive context windows, often lead to memory overload, degraded performance, or system failures.
This paper introduces a novel framework designed to enable efficient, persistent on-device AI agents. The core innovation lies in combining a structured, compressed memory system with a dynamic tool management strategy. The memory system employs two LoRA adapters—one for response generation and another for maintaining a compact, interpretable history in the form of a Context State Object (CSO). This structure captures essential interaction details while drastically reducing token usage. Simultaneously, the tool management system adopts a minimal schema serialization format and a JIT mechanism, which loads full tool definitions only when invoked, further minimizing context overhead.
The model, built upon a 3B parameter Sparse Language Model (SLM), is fine-tuned using simulated multi-turn dialogues and evaluated on a suite of complex tasks. Results demonstrate that the proposed approach reduces initial context size by over 6 times and the context growth rate by up to 25 times compared to baseline models. Despite these reductions, the system maintains or exceeds the performance of traditional models, achieving a tool call F1 score of 0.99 and high qualitative user satisfaction scores.
These findings mark a significant step toward practical, scalable on-device AI capable of handling sophisticated interactions without exceeding hardware constraints. The framework’s modularity and efficiency open avenues for deploying advanced AI assistants in privacy-sensitive, resource-limited environments, with broad implications for industry and research. Future work will focus on multimodal integration, schema adaptability, and hardware optimization, aiming to further enhance robustness and real-world applicability.
Deep Analysis
Background
边缘端AI的发展经历了从简单规则系统到深度学习模型的演变。早期如GPT-2、BERT等模型主要依赖云端计算，受限于通信延迟和隐私保护。近年来，为满足本地化需求，出现了一些小型模型（如Gemma-3、Phi-3-mini），在设备上实现推理，但受限于模型容量和上下文长度，难以支持复杂多轮交互。长上下文模型（如Longformer、RETRO）虽能处理更长序列，但在硬件资源有限的边缘设备上仍难以实现实时响应。现有方案多采用KV缓存或摘要技术，但在保持信息完整性和效率之间存在折中。本文提出的结构化记忆和JIT加载机制，为边缘端AI提供了新的解决思路，突破了上下文限制，推动了个性化智能助手的落地。
Core Problem
边缘设备上的AI代理面临两个核心挑战：一是有限的内存资源限制了上下文长度，导致对话信息易丢失或响应不连贯；二是工具Schema庞大，增加上下文负担，影响模型性能。长对话和复杂任务对模型的记忆和推理能力提出更高要求，但现有技术难以在硬件限制下实现持续、丰富的交互体验。这些问题严重制约了边缘AI的应用推广，亟需创新的上下文管理策略以提升效率与效果。
Innovation
本研究的主要创新点包括：1）基于LoRA的双重适配器架构，有效压缩对话历史，形成结构化的上下文状态对象（CSO）；2）设计极简化的工具Schema序列化格式，减少令牌消耗；3）引入按需加载（JIT）机制，仅在调用工具时加载完整Schema，极大降低上下文负载。这些创新结合模型微调和模拟多轮对话数据，显著提升边缘设备的多轮交互能力，突破传统模型在上下文长度上的限制。与现有长上下文模型相比，本文方案在硬件资源有限的环境中实现了更高的效率和持续性，为边缘AI的实用化提供了新思路。
Methodology
- �� 设计基于LoRA的双适配器系统：包括主适配器（Executor）和状态追踪器（Mem），分别负责响应生成和历史压缩；• 构建结构化的CSO，采用键值对格式，记录关键信息，确保可解释性和高效存储；• 开发极简工具Schema格式，去除冗余信息，降低令牌消耗；• 实现JIT机制：在工具调用前提供轻量工具列表，调用时加载完整Schema，减少上下文负担；• 采用模拟多轮对话数据进行微调，结合真实任务验证模型性能；• 通过多场景评估，比较不同方案的上下文增长和任务准确性。
Experiments
采用自建的多轮对话模拟数据集，涵盖多工具、多场景任务，评估模型在复杂交互中的表现。对比基线模型（如xLAM-2 3B）和不同优化方案，指标包括工具调用F1、上下文大小、响应质量。模型微调采用LoRA技术，训练数据包括多样化的工具和任务。评估在实际设备（如三星Galaxy S25）上进行，测试模型的响应速度、上下文增长和任务完成度。通过消融实验验证结构化CSO和JIT加载的贡献，确保方案的有效性和实用性。
Results
模型在复杂多轮任务中实现F1得分0.99，显著优于传统模型的0.41。初始上下文由3200 tokens降至400 tokens，减少了8倍，长对话中上下文增长速率降低10至25倍。工具调用的精确率达97%，召回率95%，整体F1超过0.98。实验还显示，结构化CSO和JIT机制在保持任务性能的同时，有效降低了上下文负载，为边缘设备提供了持续交互的可能性。这些结果验证了提出方案在实际应用中的优越性。
Applications
该技术适用于智能手机、物联网设备等边缘终端，支持本地多轮对话、工具调用和个性化服务。无需频繁连接云端，保障用户隐私，降低延迟，提升用户体验。未来可结合多模态信息处理，实现更丰富的交互场景，推动智能助手、智能家居等行业的普及。
Limitations & Outlook
模型在极端复杂或多工具场景下仍可能出现信息丢失或响应不准确的问题。JIT加载机制引入一定延迟，工具定义频繁变更时需频繁更新Schema，影响实时性。模型微调和模拟数据有限，泛化能力有待提升。未来需优化Schema更新策略，增强模型的鲁棒性和适应性。
Plain Language Accessible to non-experts
想象你在厨房做饭，厨房里有很多工具，比如刀、锅、菜板。每次做菜时，你需要用到不同的工具，但厨房空间有限，不能放太多东西。于是，你决定只带上最常用的工具，把不常用的工具放在储藏室里，只在需要时才拿出来用。这样，你就可以节省空间，做菜也更快更顺利。这个方法就像论文里的技术：用一种聪明的方法，把对话中的重要信息压缩存储，只在需要用到工具时再加载完整信息，既节省空间，又保证效率。这样，手机或其他设备就能像厨房一样，快速、顺畅地完成复杂任务，不会因为空间不够而卡顿或出错。
ELI14 Explained like you're 14
你知道吗，就像玩游戏时，你的背包里不能装太多东西，否则会变得很慢或者卡顿。为了让游戏更流畅，你会把不常用的装备放到仓库里，只在需要时再拿出来用。论文里的技术也是这样：他们设计了一种聪明的方法，把对话和工具信息压缩成小小的包裹，只在真正需要用到工具时才把完整的工具信息加载到手机里。这样，手机就能更快地记住对话内容，也能同时用很多工具，不会因为空间不够而卡顿。这个办法让手机变得更聪明、更厉害，可以陪你聊天、帮你安排事情，还能保持对话的连贯性，像个贴心的助手一样！
Abstract
On-device AI agents offer the potential for personalized, low-latency assistance, but their deployment is fundamentally constrained by limited memory capacity, which restricts usable context. This reduced practical context window creates a trade-off between supporting rich, stateful interactions with complex tool capabilities and maintaining on-device feasibility. We break this trade-off with a framework for context-efficient on-device agents, driven by three synergistic optimizations (1) a dynamic memory system using specialized LoRA adapters to distill conversational history into a compressed, and structured Context State Object; (2) a minimalist serialization format for tool schemas to minimize token overhead per tool; and (3) a just-in-time schema-passing mechanism that loads full tool definitions only upon tool selection. We instantiate this framework by adapting a 3B parameter SLM to context-efficient trajectories and rigorously evaluate it against a conventional baseline on complex user tasks. Our agent matches, or exceeds, the performance of a conventional baseline while dramatically compressing context, achieving more than a 6-fold reduction in initial system prompt context and a 10- to 25-fold reduction in context growth rate based on the interaction verbosity, demonstrating that strategic context management is key to unlocking capable and persistent on-device AI.