SKIMIX: Multi-Agent Harness-Time Scaling with Skill Mixture for Dynamic Harness Engineering
SKIMIX integrates multi-agent skill retrieval, anti-dilution routing, and adaptive evolution, boosting open-ended reasoning by 33%.
Key Findings
Methodology
The paper introduces SKIMIX, a framework combining embedding-based skill retrieval, submodular maximization for anti-dilution routing (ADR), and continuous skill evolution (ASE). Multiple agents, each with distinct skill portfolios, iteratively share and refine answers. Skills are represented as dense vectors in an embedding space, retrieved via cosine similarity. ADR employs submodular optimization to select diverse skill subsets, preventing redundancy. ASE updates skills based on recent performance, deprecating stale ones and synthesizing new skills from reasoning traces. The system operates over multiple rounds, with each agent generating solutions based on the query and previous responses, guided by adaptive stopping criteria. Experiments on six benchmarks demonstrate significant gains in open-ended tasks but limited or negative effects on multiple-choice tasks, highlighting task-dependent effectiveness.
Key Results
- On AIME, accuracy jumps from 0% to 40% with self-refinement, reaching 73.3% with three agents and 76.7% with fifteen. Similarly, on MATH-500, SKIMIX-5 achieves 72.0%, outperforming the baseline 61%. In GPQA, a multiple-choice benchmark, single-agent self-refinement achieves 88.0%, while adding more agents degrades performance, indicating task-specific effects. The second iteration yields the largest improvements across all benchmarks, with diminishing returns afterward. The non-monotonic scaling suggests optimal agent count varies by task, with three agents often outperforming larger ensembles.
- Significance: This work advances the management of large skill libraries by integrating dynamic retrieval, diversity control, and evolution, addressing skill dilution and information overload issues. It demonstrates that task characteristics critically influence multi-agent benefits, providing a pathway toward scalable, adaptable AI systems capable of complex reasoning. The framework’s ability to balance skill diversity and redundancy paves the way for more robust, efficient multi-skill AI applications in education, research, and industry, especially in open-ended reasoning scenarios.
Technical Contribution
The paper’s key technical innovations include the development of a learned skill embedding space, a submodular optimization-based anti-dilution router (ADR), and an adaptive skill evolution (ASE) mechanism. These components enable scalable, dynamic management of skill portfolios, ensuring diversity and relevance. The multi-agent iterative refinement process, guided by these modules, allows for effective answer synthesis and skill adaptation over multiple rounds. The framework guarantees performance bounds via submodular maximization and introduces a novel approach to skill ecosystem health, facilitating large-scale deployment and continuous improvement of multi-skill systems.
Novelty
This is the first systematic integration of multi-agent collaboration with dynamic skill retrieval and anti-dilution optimization, addressing the long-standing problem of skill dilution in large libraries. Unlike prior static or manually curated systems, SKIMIX employs learned embeddings and submodular optimization to automatically select and evolve skills, providing a task-aware, scalable solution. Its combination of iterative refinement, diversity control, and skill evolution represents a significant departure from existing static or model-centric approaches, filling a critical gap in multi-skill AI research.
Limitations
- The framework is primarily validated on DeepSeek-V3.2, limiting generalization to other models or architectures. Broader validation is necessary.
- The fixed skill library of 15 portfolios may not capture the full diversity needed for all tasks; scalability to larger libraries remains to be tested.
- While the theoretical basis for ADR and ASE is strong, empirical ablation studies are limited, and their individual contributions need further validation in diverse environments.
Future Work
Future directions include scaling up the skill library, refining the skill evolution process with reinforcement learning, and developing more sophisticated answer aggregation methods beyond majority voting. Additionally, exploring task-specific adaptive strategies and extending the framework to real-world applications like autonomous agents and complex decision-making systems will be crucial. Integrating multi-modal skills and improving robustness across diverse domains are also promising avenues.
AI Executive Summary
The rapid development of large language models (LLMs) has revolutionized AI-driven reasoning, but managing extensive skill libraries remains a significant challenge. Traditional systems often rely on static skill sets or manual selection, which struggle with redundancy, scalability, and adaptability. This paper introduces SKIMIX, a novel multi-agent framework designed to dynamically manage and optimize skill portfolios for complex reasoning tasks.
At its core, SKIMIX combines embedding-based skill retrieval, submodular optimization for anti-dilution routing (ADR), and continuous skill evolution (ASE). Multiple agents, each equipped with distinct skill subsets, process queries in parallel, iteratively sharing and refining solutions. The retrieval mechanism employs cosine similarity in a learned embedding space, ensuring relevant skills are prioritized. ADR uses submodular maximization to select diverse, non-redundant skill sets, preventing dilution as the skill library expands. ASE updates skills based on recent performance, deprecating ineffective ones and synthesizing new skills from observed reasoning traces.
Experiments across six benchmarks—including open-ended reasoning tasks like AIME and MATH-500, and multiple-choice benchmarks like GPQA—demonstrate the framework’s effectiveness. In open-ended tasks, SKIMIX with three agents achieves accuracy improvements of over 33% compared to single-agent self-refinement, reaching up to 76.7%. Conversely, in MCQ tasks, single-agent self-refinement remains superior, with additional agents degrading performance. The second iteration consistently yields the largest gains, with diminishing returns afterward. Notably, the number of agents does not always positively correlate with performance, emphasizing the importance of task-specific tuning.
Overall, SKIMIX offers a scalable, adaptive solution for managing large skill ecosystems, addressing key issues like skill dilution and redundancy. Its ability to dynamically select, evolve, and coordinate skills paves the way for more robust, efficient AI systems capable of tackling diverse, complex reasoning challenges. The insights gained highlight that task characteristics should guide multi-agent deployment, marking a significant step toward intelligent, scalable multi-skill AI architectures.
Deep Analysis
Background
近年来，随着大规模语言模型（LLMs）在自然语言理解和推理任务中的突破，技能库的规模不断扩大，推动多技能系统的发展。早期工作如LangChain、AutoGen等将技能作为模块，提升了系统的灵活性，但面临技能稀释和信息过载的挑战。随着技能数量的增长，冗余技能引入噪声，影响推理质量。多智能体协作、工具使用和测试时扩展（如Mixture-of-Agents、Tool-Use Mixture）为解决方案提供了启示，但缺乏系统化的动态管理机制。本文提出SKIMIX，旨在实现技能的高效检索、多样性控制和动态演化，推动多技能系统的可扩展性和适应性。
Core Problem
现有多技能系统在技能选择、组合和维护方面存在瓶颈。技能库不断扩大导致冗余和信息过载，影响推理效率和准确性。静态管理难以应对任务变化和技能演化需求，限制了多技能系统的应用范围。如何在保证技能多样性的同时，避免技能稀释和噪声引入，成为核心难题。此外，现有方法多为静态配置，缺乏动态调节机制，难以实现高效、可扩展的多技能推理。
Innovation
本文提出三大创新：
1) 动态技能注册（DSR），通过学习嵌入空间实现技能的快速检索和动态更新；
2) 子模反稀释路由（ADR），利用子模最大化控制技能多样性，避免冗余；
3) 持续技能演化（ASE），根据性能反馈动态调整技能集，保持系统的适应性。结合多智能体的迭代优化策略，实现技能的自动选择、演化和协作，突破静态技能库的限制，显著提升推理能力和系统鲁棒性。
Methodology
- �� 构建全局技能注册表（S），每个技能通过句子编码模型学习获得嵌入向量（ei），支持注册、检索和演化。• 多智能体系统，每轮选择不同的技能组合（Pj），通过余弦相似度进行技能检索。• ADR利用子模最大化（最大化相关性与多样性）选择技能子集，避免冗余，保证多样性。• 每轮多智能体独立生成答案，信息通过消息传递机制共享，逐步优化答案。• 采用早停策略，根据边际收益与成本比判断是否继续迭代。• 最终通过投票或融合策略输出答案。• 实验在六个推理任务（AIME、GPQA、HLE、MMLU-Pro、MATH-500、BBH）上验证，比较单智能体、自我优化与SKIMIX多智能体配置的性能差异。
Experiments
使用DeepSeek-V3.2模型，设置不同智能体数（1、3、5、15），在六个推理任务上评估准确率、覆盖率。每个任务采用官方标准指标，结合多轮迭代、技能配置变化进行对比。对比包括单智能体自我优化、多智能体SKIMIX（不同规模）以及不同轮次的性能变化。实验还进行消融分析，验证技能检索、反稀释和演化机制的贡献。结果显示，开放式推理任务中多智能体合作显著优于单智能体，而在多项选择题中效果有限甚至倒退。
Results
在AIME任务中，三智能体SKIMIX准确率达73.3%，远超单智能体40%，提升83%。在MATH-500，SKIMIX-5达72.0%，优于自我优化61%。GPQA中，单智能体自我优化达88.0%，多智能体反而降低性能，说明任务类型影响效果。第二轮优化带来最大提升，第三轮效果递减。智能体数量非单调，三智能体在部分任务优于五或十五智能体，强调任务特性的重要性。多智能体配置在开放式任务中覆盖率高于最终准确率，显示答案选择仍是瓶颈。
Applications
该框架适用于复杂推理、教育辅导、科研辅助等场景，能动态管理技能库，提升系统应变能力。企业可用其优化多技能问答系统，提升用户体验。未来结合强化学习与更大规模技能库，将推动智能系统在自动化决策、知识推理等领域的应用。
Limitations & Outlook
当前实验主要基于DeepSeek-V3.2，模型多样性不足，需验证在不同模型架构上的效果。技能库规模有限，未充分探索更大规模的技能组合。ADR与ASE机制虽有理论基础，但未进行充分消融验证，未来应验证其在更复杂环境中的表现。此外，答案融合策略仍有提升空间，需开发更鲁棒的集成方法。
Plain Language Accessible to non-experts
想象一个厨房里有很多厨师，每个厨师擅长不同的菜系。有时候，单个厨师做菜可能不够快或不够好，但如果让几个厨师合作，他们可以互相帮忙，做出更美味的菜肴。这个系统就像是让厨师们轮流合作，分享彼此的技巧，避免重复和无用的菜肴。通过不断调整厨师的技能和合作方式，厨房的效率和菜品质量都能不断提升。这个方法就像是厨房里的智能助手，学会了哪些厨师擅长哪些菜，什么时候该让他们合作，什么时候该让某个厨师独立工作。这样一来，不仅菜做得更好，还能节省时间和资源。它的核心思想是让不同的厨师（技能）合作得更聪明，避免重复和无用的工作，最终让厨房变得更高效、更有创意。
ELI14 Explained like you're 14
想象你在学校里有很多朋友，每个人都擅长不同的科目。有时候，你需要帮忙做数学题，有时候又要写作文。如果你只找一个朋友帮忙，可能会花很多时间，而且不一定能找到最好的答案。但如果你让几个朋友轮流帮忙，每个人都用自己的强项出主意，最后你就能更快更好地完成任务。这就像是让多个“智能朋友”合作，每个朋友都带着自己的技能，轮流讨论、改进答案。刚开始，大家都试着出主意，答案可能还不完美，但经过几轮讨论，答案会变得越来越好。有时候，加入太多朋友反而会让讨论变得混乱，所以最好的办法是找到合适的朋友数量，让合作既高效又有趣。这种方法可以让你在学习和解决难题时变得更聪明、更快。它的秘密在于合理安排朋友的角色，让每个人都发挥最大作用，避免重复和无用的建议，最终得到最棒的答案。
Glossary
Skill Embedding (技能嵌入)
将技能描述转化为向量空间中的密集向量，用于相似度检索和匹配。技术上采用句子编码模型，便于快速查找相关技能。
在SKIMIX中，用于技能检索和动态更新，确保相关性和多样性。
Anti-Dilution Routing (反稀释路由)
通过子模最大化控制技能集合的多样性，避免冗余技能引入噪声，提升系统效率。采用子模优化算法实现。
在技能选择阶段，用于筛选最具代表性和多样性的技能子集。
Skill Mixture (技能混合)
多智能体协作框架，通过不同技能组合实现任务解决，结合动态检索、反稀释和演化机制。
核心创新，用于提升多技能系统的推理能力和适应性。
Open Questions Unanswered questions from this research
- 1 如何在更大规模技能库中保持技能多样性与效率的平衡仍未充分解决，未来需探索更高效的检索与管理机制。
- 2 多智能体合作中，如何优化答案融合策略以减少噪声，提高最终准确率，是一个开放问题。
- 3 技能演化机制的长远影响及其在不同任务中的适应性仍需深入研究，尤其在动态环境下的表现。
Applications
Immediate Applications
智能问答系统
结合SKIMIX实现多技能问答平台，提升复杂问题的解答效率和准确性，适用于客服、教育等场景。
科研辅助工具
支持科研人员在复杂推理和数据分析中动态调配技能，提升研究效率和创新能力。
Long-term Vision
智能自主系统
未来可发展为自主学习与演化的多技能智能体，广泛应用于自动化决策、机器人等领域，推动AI的自主适应能力。
Abstract
AI agents increasingly rely on large skill libraries, but selecting, combining, and maintaining skills remains difficult. We propose SKIMIX, a multi-agent framework in which agents with different skill portfolios collaborate through iterative refinement. SKIMIX combines embedding-based skill retrieval, submodular anti-dilution routing, and adaptive skill evolution. Across six reasoning benchmarks, multi-agent collaboration substantially improves open-ended mathematical reasoning but offers limited or negative gains on multiple-choice tasks. Agent-count scaling is non-monotonic, and most improvements arise during the first refinement round. These results show that task characteristics determine whether skill-level ensembles help and provide practical guidance for scalable agent design.