AgentFlux: Decoupled Fine-Tuning & Inference for On-Device Agentic Systems
AgentFlux uses decoupled fine-tuning with LoRA adapters, boosting local tool-calling accuracy by 46% on MCP-Bench.
Key Findings
Methodology
This paper introduces decoupled fine-tuning, leveraging LoRA (Low-Rank Adaptation) to separately train adapters for tool selection and argument generation. By splitting these subtasks and applying distinct loss masks, the approach enhances task-specific performance. Synthetic data generated via gpt-4o supports robust training. The framework incorporates hierarchical orchestration, dynamically loading adapters to manage extensive toolsets efficiently. Experiments on MCP-Bench show a 46% accuracy increase for Qwen-2.5-7B, outperforming similar-sized and larger models, demonstrating the method’s effectiveness in on-device agent systems.
Key Results
- Decoupled fine-tuning improves tool call accuracy of Qwen-2.5-7B by 46%, surpassing baseline models (16%) and traditional fine-tuning (~20%). Hierarchical scheduling reduces context length, maintaining high accuracy (>70%) and halving inference latency. Across multiple toolsets, the approach demonstrates robustness and scalability, outperforming other local models and matching or exceeding larger models in tool invocation tasks.
- In environments with large toolsets, the method maintains stable performance, effectively limiting context size and improving decision accuracy. Ablation studies confirm that task decoupling and hierarchical routing are key to performance gains. The dynamic adapter loading mechanism enables efficient inference, suitable for resource-constrained devices.
- Compared to prior methods like prompt tuning or end-to-end fine-tuning, decoupled fine-tuning with hierarchical scheduling offers superior accuracy and speed, especially in complex, multi-tool scenarios. It sets a new standard for local agent orchestration performance.
Significance
This work addresses critical bottlenecks in deploying privacy-preserving, on-device intelligent agents. By significantly improving tool-calling accuracy, it enables practical applications in edge devices, reducing reliance on cloud APIs and associated costs. The modular architecture facilitates scalability and robustness, paving the way for autonomous AI systems in sensitive environments. The combination of task decoupling and hierarchical management introduces a new paradigm in model fine-tuning and inference, with broad implications for industry and research, fostering more efficient, private, and scalable AI solutions.
Technical Contribution
The core innovation lies in the decoupling of tool selection and argument generation tasks, each trained with dedicated LoRA adapters. Synthetic data generation via gpt-4o ensures high-quality training samples. The hierarchical orchestration framework effectively manages large toolsets by dividing decision layers, reducing context size and improving accuracy. The dynamic adapter loading mechanism, powered by vLLM, enables efficient on-device inference. These techniques collectively advance the state-of-the-art in local agent orchestration, offering a scalable, accurate, and privacy-preserving solution.
Novelty
This is the first work to combine decoupled fine-tuning with hierarchical toolset management for local models in agent systems. Unlike prior approaches that treat tool calling as a monolithic task, this method decomposes it into specialized sub-tasks, enabling targeted optimization. The integration of synthetic data, task-specific adapters, and multi-layer scheduling constitutes a novel framework that significantly outperforms existing methods in accuracy and efficiency, especially under resource constraints.
Limitations
- The approach relies heavily on synthetic data quality; biases or errors in generated data may limit generalization. In extremely large toolsets, hierarchical routing complexity could increase, affecting performance. Hardware constraints still pose challenges for deploying multiple adapters on low-end devices. Further research is needed to optimize multi-task training and multi-modal inputs for broader applicability.
Future Work
Future directions include integrating reinforcement learning to optimize tool selection policies, expanding multi-modal capabilities, and enhancing adaptive scheduling strategies. Improving data generation methods to reduce biases and increase diversity is also crucial. Extending the framework to support more complex multi-task scenarios and real-world deployments will accelerate the adoption of privacy-preserving, autonomous edge AI systems.
AI Executive Summary
The deployment of large language models (LLMs) as autonomous agents has revolutionized task automation, yet their performance heavily depends on effective tool calling capabilities. Traditional fine-tuning methods struggle to optimize both tool selection and argument generation simultaneously, especially in resource-constrained, on-device environments. This paper introduces AgentFlux, a novel framework that leverages decoupled fine-tuning with LoRA adapters to address these challenges. By separating the two core subtasks—tool selection as a classification task and argument generation as a structured output—the approach achieves targeted optimization. Synthetic data generated via gpt-4o underpins robust training, while hierarchical orchestration manages large toolsets efficiently by dividing decision layers. Extensive experiments on MCP-Bench demonstrate that the Qwen-2.5-7B model, after decoupled fine-tuning, improves tool call accuracy by 46%, outperforming comparable models and even larger models in many cases. This significant performance boost enables local models to operate effectively as autonomous agents, preserving privacy and reducing costs. The layered architecture ensures scalability and robustness, making it suitable for deployment on consumer-grade hardware. The research opens new avenues for privacy-preserving, scalable AI systems, with potential applications spanning personal assistants, industrial automation, and beyond. Future work will focus on multi-modal inputs, reinforcement learning integration, and broader real-world deployment, promising a future where intelligent agents can operate seamlessly on edge devices with high accuracy and efficiency.
Deep Analysis
Background
近年来，随着GPT-3、Llama等大型预训练模型的崛起，智能代理系统逐渐成为研究焦点。早期工作如OpenAI的GPT-4和Anthropic的Claude主要关注模型规模和预训练策略，但在工具调用方面仍存在性能瓶颈。标准协议如MCP推动了模型与外部工具的集成，但本地模型在工具选择和参数生成方面表现不足。传统微调和prompt调优虽有一定改善，但在长上下文和大规模工具集环境中效果有限。边缘设备对隐私和成本的需求促使研究转向本地模型部署，然而性能差距依然明显。本文在此背景下，提出解耦微调和层次调度，旨在突破本地模型在工具调用中的瓶颈。
Core Problem
核心问题在于本地模型在工具调用中的表现不足，尤其是在工具选择和参数生成两个环节。工具选择作为分类任务，受限于上下文长度和注意力机制，难以准确识别合适工具。参数生成要求结构化输出，模型在复杂参数和长描述中易出错。长描述和多工具集增加了模型负担，导致调用准确率低、延迟高。这限制了在隐私敏感和成本受控场景中的应用，亟需突破。
Innovation
本研究的创新点包括：1）引入解耦微调，将工具选择与参数生成任务分离，利用LoRA适配器实现任务专用化，提升性能；2）通过gpt-4o自动生成多样化训练数据，增强模型泛化能力；3）提出层次化调度，将工具集划分为子集，减少上下文长度，提升决策效率。这些创新区别于传统端到端微调，强调任务分解和模块化训练，显著改善本地模型在工具调用中的表现。
Methodology
- �� 利用gpt-4o自动生成包含所有工具和参数场景的高质量训练数据。
- �� 通过模拟工具调用轨迹，提取训练样本，训练两个LoRA适配器：工具选择器和参数生成器。
- �� 工具选择器负责分类任务，预测下一步调用的工具名，采用损失掩码只计算工具名部分。
- �� 参数生成器针对每个工具，预测具体参数，损失只在参数部分。
- �� 采用分层调度：先用基础模型进行高层工具集选择，再加载子集内的工具选择器，减少上下文复杂度。
- �� 动态加载适配器，结合vLLM实现高效推理，支持多工具集环境。
- �� 在MCP-Bench上进行大规模评估，验证准确率和延迟改善。
Experiments
采用MCP-Bench中的文件系统工具集，训练数据由gpt-4o生成，模型微调后在50个任务上测试。对比传统微调和prompt调优效果，验证解耦微调的性能提升。引入层次调度机制，评估不同工具集规模对模型性能的影响。指标包括工具调用准确率（ToolFit）和推理延迟。实验还包括不同模型（Qwen-2.5-7B、Llama-8B等）在相同策略下的表现对比，验证方法的普适性和优越性。
Results
解耦微调后，Qwen-2.5-7B模型在MCP-Bench上的工具调用准确率提升46%，显著优于未微调模型（16%）和传统微调（约20%）。层次调度有效缩短上下文长度，保持高准确率（超过70%），且推理延迟降低一半以上。多工具集环境下，模型表现稳定，适应性强。对比其他本地模型，解耦微调方案在工具调用精度和响应速度方面均优越，验证了其在实际边缘场景中的应用潜力。
Applications
该技术适用于边缘设备上的智能助手、自动化办公、隐私敏感场景等。只需在本地部署微调模型，无需云端交互，保障数据隐私。可广泛应用于企业内部自动化、个人助理和工业控制等领域。未来结合多模态输入和强化学习，将实现更智能、更自主的边缘AI系统，推动行业数字化转型。
Limitations & Outlook
当前方法依赖合成数据的质量，若生成偏差可能影响模型泛化能力。在极端大规模工具集下，层次调度复杂度可能增加，影响性能。硬件限制仍是推广难题，低端设备上多适配器加载存在难度。未来需优化多任务训练策略和多模态融合技术，以实现更广泛应用。
Plain Language Accessible to non-experts
想象你在厨房做饭，厨房里有很多不同的工具，比如刀、锅、碗碟。每次做菜时，你需要先决定用哪个工具（比如用刀切菜），然后用它完成具体任务（比如切胡萝卜）。如果你每次都要重新思考用哪个工具，效率会很低。现在，科学家们让电脑学会像你一样，先快速决定用哪个工具，然后再用专门的“助手”帮忙准备工具的具体参数（比如刀的角度、大小）。他们还设计了一个“厨房管理系统”，可以把工具分类成不同的组（比如切菜组、炒菜组），每组有自己的助手，这样每次只需要考虑一小部分工具，效率就大大提高。这就像你在厨房里有不同的助手帮你，既快又准。这个方法让电脑在本地就能像专业厨师一样，快速、准确地调用各种工具，做出复杂的菜肴，既保护隐私，又节省成本。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，你需要用很多不同的道具，比如剑、魔法棒、药水。每次打怪或完成任务，你都得选对道具，还得用正确的方法用它们。这很难对吧？科学家们也遇到类似的问题，他们让电脑学会像你一样，先决定用哪个道具，然后再用专门的“助手”帮忙准备好用的参数，比如用多大力气挥剑，或者魔法棒的魔法强度。为了让电脑更聪明，他们还把所有的道具分成不同的类别，比如武器组、魔法组，这样每次只需要考虑一小部分，就能更快做出决定。这就像你在游戏里有不同的队友帮你，每个队友专门负责一类任务，合作起来就特别快。这个方法让电脑在本地就能像高手一样，快速、准确地调用各种工具，完成复杂的任务，不用联网，也能保护你的隐私。是不是很酷？
Glossary
LoRA (Low-Rank Adaptation)
一种在预训练模型基础上进行参数微调的技术，通过低秩矩阵调整模型参数，提升微调效率。
用于训练工具选择和参数生成的适配器，提升模型性能。
层次调度 (hierarchical orchestration)
一种多层次的调度策略，将工具集划分为子集，先进行高层决策，再细化选择，提高效率。
用于管理大量工具，减少上下文长度，优化模型响应。
MCP (Model Context Protocol)
一种标准协议，用于定义语言模型与外部工具的交互接口。
本文中用于工具调用的标准框架。
合成数据 (synthetic data)
由模型自动生成的训练样本，用于增强微调数据集的多样性。
用于训练工具选择和参数生成适配器。
ToolFit指标
衡量模型工具调用准确率的标准。
在MCP-Bench评估中的关键指标。
Open Questions Unanswered questions from this research
- 1 如何在极端长上下文环境中持续提升模型的工具调用性能，特别是在多模态、多任务场景下，仍需深入研究。
- 2 多工具集、多任务调度的效率和鲁棒性仍是未来研究重点，结合强化学习等技术可能带来突破。
Applications
Immediate Applications
边缘设备智能助手
在手机、笔记本等设备上部署微调模型，支持隐私保护的自动化任务，无需云端依赖。
工业自动化控制
在工厂或现场设备中实现自主调度和工具调用，提高效率和安全性。
Long-term Vision
自主边缘AI系统
实现完全自主、智能的边缘设备，支持多模态输入、多任务协同，推动智能制造和智慧城市发展。
Abstract
The deployment of Large Language Models (LLMs) as agentic orchestrators has revolutionized task automation, but the need for privacy-preserving, cost-effective solutions demands on-device inference capabilities. However, local LLMs consistently underperform compared to frontier models in tool calling scenarios, struggling with both tool selection from large tool sets and accurate argument generation for complex parameter structures. We introduce a methodology that disaggregates a tool-calling task into two distinct subtasks: tool selection and argument generation. We propose "decoupled fine-tuning", a novel post-training approach that employs LoRA fine-tuning to create dedicated LoRA adapters for tool selection and tool-specific argument generation using separate loss masking for each of the subtasks. Furthermore, we present AgentFlux, an inference framework that leverages the LoRA adapters created using decoupled fine-tuning to perform efficient agent orchestration with the help of local models on end-user devices. AgentFlux decomposes the tool-call generation step into tool selection and argument generation, and dynamically loads the corresponding LoRA adapters to generate tool calls. Additionally, AgentFlux implements hierarchical orchestration to restrict the number of tools required for tool selection. Our experiments on the MCP-Bench benchmark demonstrate that the Qwen-2.5-7B model trained using decoupled fine-tuning improves the tool calling accuracy of the base model by 46%, and outperforms other local reasoning, non-reasoning and fine-tuned models of similar size in all cases, and models that are 2x larger, in most cases.