Klear-AgentForge: Forging Agentic Intelligence through Posttraining Scaling
Introduces Klear-Qwen3-AgentForge, combining supervised fine-tuning and multi-turn RL on Qwen3-8B, achieving state-of-the-art multi-task agentic performance.
Key Findings
Methodology
The paper develops a two-stage training pipeline integrating synthetic data-driven supervised fine-tuning (SFT) and multi-turn reinforcement learning (RL) based on the GRPO algorithm. It constructs multi-domain environments supporting tool invocation and coding tasks, employing a granular reward mechanism to address reward sparsity. The approach leverages environment simulation, model merging, and asynchronous architecture to enhance stability and scalability, resulting in a high-performance agentic model that surpasses comparable models in multiple benchmarks.
Key Results
- Klear-Qwen3-AgentForge-8B achieves 71.5 and 56.7 scores on tool use and coding benchmarks, outperforming similarly sized models and approaching larger models like GPT-4.1; on BFCL v3, τ-bench, and SWE-bench, it demonstrates superior multi-task generalization and robustness, validating the effectiveness of multi-stage training.
- Scaling model parameters and training data significantly improves performance, especially in multi-turn reasoning and environment interaction tasks, with notable gains in stability and accuracy.
- The introduction of fine-grained rewards and asynchronous training architecture effectively mitigates reward sparsity and enhances training efficiency, leading to more stable convergence.
Significance
This work addresses the critical bottleneck in open-source development of multi-task, multi-turn agentic models, providing a comprehensive, fully open-source training pipeline. It advances the state-of-the-art in multi-domain AI agents, with broad implications for industrial automation, software engineering, and autonomous systems, laying a foundation for future generalist AI agents.
Technical Contribution
The paper innovates by integrating synthetic data-driven SFT with multi-turn RL, employing a granular reward mechanism and asynchronous architecture. It introduces a model merging strategy for multi-task knowledge fusion, significantly improving training stability and performance over traditional single-task or single-turn methods. The approach also enhances environment interaction capabilities, setting new benchmarks for open-source multi-task agents.
Novelty
This is the first comprehensive system combining synthetic data-based SFT and multi-turn RL for multi-task agent training, with an effective model merging strategy. It demonstrates that open-source models can achieve performance comparable to proprietary large models, representing a significant breakthrough in scalable, multi-domain AI agent development.
Limitations
- The training process relies heavily on high-quality synthetic data, which incurs substantial generation costs and may limit scalability. In complex tasks, generalization still faces challenges due to domain-specific biases. The architecture demands significant computational resources, restricting rapid scaling and deployment. Future work should focus on data efficiency, robustness, and reducing resource requirements.
Future Work
Future directions include optimizing synthetic data generation, refining multi-task merging strategies, and exploring more efficient environment simulation. Enhancing self-supervised learning and continual adaptation will further improve autonomous capabilities, moving toward more general, adaptable AI agents capable of handling increasingly complex real-world scenarios.
AI Executive Summary
The rapid evolution of large language models (LLMs) has revolutionized natural language processing, yet their ability to perform complex, multi-task, multi-turn interactions remains limited. Traditional models excel at single-turn predictions but struggle with sustained reasoning, environment interaction, and tool utilization. To bridge this gap, the authors introduce Klear-Qwen3-AgentForge, a novel training framework built upon the open-source Qwen3-8B model, designed to develop highly capable agentic models.
This framework combines synthetic data-driven supervised fine-tuning (SFT) with multi-turn reinforcement learning (RL). The SFT stage leverages a large corpus of synthetic and real data, including tool-use and coding trajectories, to instill foundational skills. The RL stage employs a granular reward mechanism, inspired by GRPO, to guide the model through multi-step reasoning and environment interactions, effectively addressing reward sparsity. An asynchronous training architecture further enhances efficiency, enabling scalable multi-task learning.
Experimental results demonstrate that the resulting model, Klear-Qwen3-AgentForge-8B, achieves state-of-the-art performance on multiple benchmarks, including BFCL v3, τ-bench, and SWE-bench, surpassing comparable models and approaching larger proprietary systems like GPT-4.1. The model exhibits robust multi-task generalization, with performance improvements correlating with increased model size and training data volume. Ablation studies confirm the importance of reward granularity and architecture design.
This work significantly advances open-source AI development, providing a complete, scalable pipeline for training versatile, multi-task agentic models. Its implications span industrial automation, software engineering, and autonomous systems, fostering the creation of more intelligent, autonomous agents capable of complex reasoning and environment interaction. Future efforts will focus on reducing data costs, enhancing robustness, and extending capabilities to more complex, real-world tasks.
Deep Analysis
Background
近年来，LLMs在自然语言理解和生成中取得巨大突破，但多任务、多轮交互能力仍受限。早期工作如GPT-3、T5、BERT主要关注单轮预测，难以应对复杂环境中的连续决策。ReAct、Toolformer等模型引入工具调用和推理机制，提升了交互能力，但在奖励稀疏、环境模拟和训练效率方面仍存在瓶颈。开源社区对高效、多任务、多轮智能体的需求不断增长，推动了多阶段训练体系的研发，旨在实现更强的自主交互能力。
Core Problem
核心问题在于如何系统性地训练具备多任务、多轮交互能力的模型。传统方法多依赖单一训练策略，难以兼顾不同任务的特性，且奖励稀疏导致学习效率低。如何设计统一的训练流程，结合合成数据和环境模拟，提升模型的泛化和适应能力，是当前亟待解决的难题。此外，模型在多任务融合、环境交互的稳定性和效率方面仍存在瓶颈。
Innovation
本文的创新点包括：1）提出结合合成数据的监督微调（SFT）策略，丰富模型基础知识；2）设计多轮强化学习（RL）框架，解决奖励稀疏问题，增强环境交互能力；3）采用异步架构提升训练效率，减少GPU空闲时间；4）引入模型合并策略，有效融合多任务知识。这些创新共同推动了多任务、多轮智能体的研发，突破了以往单一任务或单轮训练的限制。
Methodology
- �� 构建多域环境，支持工具调用和编码任务，定义行动空间包括生成和调用工具。
- �� 利用合成数据和真实数据，采用多轮提示生成轨迹，筛选高质量样本。
- �� 通过SFT对模型进行基础微调，增强任务相关能力。
- �� 设计多轮RL训练，采用GRPO目标，结合细粒度奖励机制，缓解奖励稀疏。
- �� 实现异步架构，分离采样和训练过程，提高效率。
- �� 采用模型合并策略，将不同任务的模型参数融合，提升多任务性能。
Experiments
在BFCL v3、τ-bench和SWE-bench等多个基准上评估模型性能，比较不同规模和训练策略的效果。使用参数扩展、数据扩充和奖励机制调优，进行消融实验验证各组件贡献。指标包括任务成功率、平均交互轮次和模型泛化能力。通过与GPT-4.1等大模型的对比，验证模型在多任务环境中的优越性。
Results
模型在工具调用任务中达到了71.5的性能分数，编码任务56.7，均优于同规模模型。多任务评测中表现出色，验证了多阶段训练的有效性。参数和数据扩展显著提升性能，奖励机制和异步架构增强训练稳定性。模型在不同任务间表现出良好的泛化能力，显示出强大的多任务适应性。
Applications
该模型可应用于自动化编码、智能助手、环境交互和工具调用等场景，适合工业自动化、软件开发和智能客服等行业。其多任务能力和环境适应性，为实现自主智能体提供技术基础。未来可结合实际环境，部署在机器人、自动驾驶和智能制造中，推动行业智能升级。
Limitations & Outlook
模型训练依赖大量高质量合成数据，数据生成成本较高，且在极端复杂任务中仍存在泛化不足。多任务融合可能引入任务干扰，影响某些任务性能。硬件资源需求大，限制了更大模型的快速训练和部署。未来需优化数据效率和模型鲁棒性。
Plain Language Accessible to non-experts
想象一个厨房里有许多厨师（模型），他们不仅要知道怎么做菜，还要会用各种工具，比如刀、锅、调料等。有的菜需要多次尝试和调整，厨师要不断试错，学会用不同的工具，最终做出美味的菜肴。这个研究就像教厨师们如何更聪明、更快地用工具，做出各种菜。通过反复练习和奖励机制，厨师们变得越来越厉害，能应对不同的菜谱和厨房环境。这就像让AI学会多任务、多轮交互，变得更智能、更灵活。
ELI14 Explained like you're 14
想象你在学校里学做菜，你不仅要知道怎么做一道菜，还要学会用不同的厨具，比如刀、锅、调料瓶。有时候，你需要试几次才能做得好，还要学会用不同的工具帮忙，比如让机器人帮你切菜或者倒汤。这个研究就像是在教AI怎么像你一样，学会用各种工具，做出不同的菜。它通过不断练习和奖励，让AI变得更聪明，能应对很多不同的任务，比如写代码、用工具，甚至解决问题。这样，未来的AI就能像一个多才多艺的厨师，帮你完成各种复杂的事情！
Abstract
Despite the proliferation of powerful agentic models, the lack of critical post-training details hinders the development of strong counterparts in the open-source community. In this study, we present a comprehensive and fully open-source pipeline for training a high-performance agentic model for interacting with external tools and environments, named Klear-Qwen3-AgentForge, starting from the Qwen3-8B base model. We design effective supervised fine-tuning (SFT) with synthetic data followed by multi-turn reinforcement learning (RL) to unlock the potential for multiple diverse agentic tasks. We perform exclusive experiments on various agentic benchmarks in both tool use and coding domains. Klear-Qwen3-AgentForge-8B achieves state-of-the-art performance among LLMs of similar size and remains competitive with significantly larger models.