StarHarness: Evolving Harnesses with Stratified Search for Enterprise Environments
StarHarness employs stratified search to evolve environment-specific harnesses, boosting performance by 20-35 points without changing model weights.
Key Findings
Methodology
StarHarness utilizes a stratified search approach, dividing tasks based on failure modes to build a compact evolution pool. It separates proposer-visible search tasks from hidden selection tasks, reserving unseen tasks for generalization testing. The core involves failure-mode-based task partitioning, combined with tree search and hill climbing to optimize scheduling strategies, all while keeping model weights fixed. The process includes proposal, validation, and evaluation stages, with persistent memory tracking candidate modifications. This approach was tested on ITBench SRE, EnterpriseOps-Gym ITSM, and AutomationBench Finance, achieving 20-35 percentage point improvements in overall performance, with transferability across models like GPT and Qwen.
Key Results
- On ITBench SRE, success rate increased from 40% to 75%, interaction steps reduced by 25, and costs decreased by 17%.
- In EnterpriseOps-Gym, success rate rose from 23.3% to 43.7%, tool calls halved, and costs cut by 53%.
- AutomationBench scores improved from 57.1% to 83.2%, with fewer interaction rounds and guardrail violations dropping from 20 to 4.
Significance
This work addresses the persistent mismatch between fixed models and complex enterprise environments by systematically evolving external scheduling strategies. It significantly enhances task success, operational efficiency, and safety, offering a practical solution for deploying robust enterprise AI agents. The approach reduces reliance on costly model fine-tuning and enables rapid adaptation to environment changes, marking a step forward in scalable, flexible AI deployment in industry settings.
Technical Contribution
StarHarness introduces a stratified, failure-mode-based task partitioning framework combined with a hybrid tree search and hill climbing algorithm. It leverages external scheduling to optimize task execution without modifying model weights, providing a scalable, environment-adaptive approach. Its key innovation lies in externalizing the scheduling logic, enabling systematic search and generalization, which surpasses traditional fine-tuning methods and offers theoretical guarantees of performance improvements.
Novelty
This study is the first to systematically incorporate failure-mode-based stratification and external scheduling search for environment adaptation in fixed-model settings. Unlike prior work focusing solely on prompt tuning or internal fine-tuning, StarHarness externalizes the scheduling process, enabling targeted, efficient optimization. Its combination of tree search and hill climbing for task strategy evolution is a novel contribution that significantly improves robustness and transferability.
Limitations
- The method relies heavily on accurate failure mode identification; ambiguous or rapidly changing failure patterns may limit effectiveness.
- Computational costs can be high due to repeated validation and evaluation, especially with large task sets or multiple models.
- Current validation is primarily on simulated enterprise environments; real-world deployment may face unforeseen variability and security issues.
Future Work
Future directions include integrating reinforcement learning to co-evolve harness strategies and model weights, enabling deeper environment adaptation. Exploring meta-learning for rapid strategy updates, and deploying in real enterprise systems for long-term stability, are promising avenues. Additionally, developing multi-model collaborative scheduling could further enhance robustness and scalability.
AI Executive Summary
In enterprise settings, AI agents must operate amidst complex tool interfaces, domain conventions, and operational constraints. Traditional approaches rely heavily on fine-tuning large language models, which often struggle to adapt quickly to environment changes, leading to performance bottlenecks. StarHarness offers a novel solution by externalizing the task scheduling process, employing a stratified search strategy that partitions tasks based on failure modes. This approach constructs a compact evolution pool, enabling targeted strategy optimization without modifying the underlying model weights.
The core innovation lies in combining tree search with hill climbing, allowing the system to explore multiple hypotheses and refine task execution strategies iteratively. By separating proposer-visible search tasks from hidden selection tasks, the method ensures that improvements are both effective and generalizable. Extensive experiments across ITBench SRE, EnterpriseOps-Gym, and AutomationBench demonstrated that the evolved harnesses outperform default configurations by 20-35 percentage points, with notable reductions in interaction steps, costs, and failure rates.
Remarkably, these improvements transfer seamlessly to unseen tasks and different models, including Qwen and other GPT variants, without re-evolution. Trace analysis reveals that the key to success is repairing interface issues, formalizing environment conventions, and encoding operational knowledge, which collectively compress the search space and reduce false positives.
This work signifies a shift towards externalized, strategy-based adaptation for fixed models, offering a scalable, practical pathway for deploying robust AI agents in complex enterprise environments. Future research aims to integrate reinforcement learning for joint optimization of harness strategies and model weights, further enhancing adaptability and efficiency, ultimately paving the way for autonomous, self-improving enterprise AI systems.
Deep Analysis
Background
随着大规模语言模型（LLM）在自动化、决策支持等企业应用中的不断普及，如何让模型更好地适应复杂多变的企业环境成为研究焦点。早期工作如GPT-4应用主要集中在微调模型参数以提升特定任务性能，但在面对状态依赖、工具接口繁杂、环境知识丰富的实际场景时，效果有限。近年来，调度策略优化、提示工程和微调方法不断发展，代表性工作包括Meta-Harness、Prompt Tuning等，旨在提升模型的任务适应性。然而，这些方法多依赖模型参数调整，难以应对环境持续变化带来的挑战。企业环境中的任务具有状态依赖、工具接口复杂、操作知识丰富等特点，传统微调难以全面适应。为此，研究逐步转向外部调度策略、任务划分和策略演化，以提升模型在实际应用中的鲁棒性和效率。
Core Problem
企业环境中的AI代理需要在复杂的工具接口、多样的任务场景下保持高效表现。现有微调方法难以快速适应环境变化，且调度策略设计缺乏系统性，导致模型与环境偏差持续存在。具体瓶颈包括：任务调度缺乏系统化划分、搜索空间庞大导致效率低下、模型微调成本高且难以迁移。如何在保持模型固定的基础上，通过调度策略的优化实现环境适应，成为亟待解决的问题。这不仅关系到企业自动化的效率，也影响模型的安全性和稳定性。
Innovation
StarHarness的核心创新在于：1）引入基于失败行为的任务分层，将任务划分为不同失败模式，提升搜索效率；2）结合树搜索与爬山算法，平衡探索与利用，优化调度路径；3）设计任务划分与未见任务保留机制，增强模型的泛化能力。这些创新使得调度策略在保持模型权重不变的前提下，有效适应不同企业环境。其独特之处在于：通过外部调度器对任务调度进行系统化演化，减少了微调的复杂性和成本，为企业AI代理提供了更灵活的解决方案。
Methodology
- �� 任务划分：基于失败模式、任务得分和验证通过率，将任务划分为搜索集、选择集和未见集。• 构建演化池：从全部任务中采样，确保失败行为分布均衡，避免过拟合。• 搜索策略：采用树搜索探索多假设路径，结合爬山算法进行局部优化。• 迭代演化：在提案、验证、评估三阶段循环中，利用持久存储追踪候选修正，确保搜索的连续性。• 保护机制：限制候选变更范围，避免硬编码答案或任务特定信息。• 评估：在未见任务上进行最终验证，确保泛化能力。• 迁移：在不同模型（GPT、Qwen）上测试演化效果，验证其泛化能力。
Experiments
采用ITBench SRE、EnterpriseOps-Gym和AutomationBench三大企业场景，分别测试根因分析、ITSM流程和财务自动化任务。对比默认调度策略与演化策略的性能差异，指标包括任务成功率、交互轮次、成本和误诊率。使用GPT-5.4（中等推理能力）作为基础模型，结合不同模型（Qwen3.6、GPT-5.5）进行迁移验证。超参数包括：演化轮次（4-12次）、任务划分比例（约50%）、搜索深度。还设计了消融实验，验证失败行为划分和搜索策略的贡献。
Results
演化后，ITBench成功率由40%提升至75%，成本降低17%；EnterpriseOps-Gym成功率由23.3%提升至43.7%，工具调用次数减半，成本降低53%；AutomationBench得分由57.1%提升至83.2%，轮次减少4.37，guardrail违规显著下降。迁移测试中，模型从Qwen3.5到GPT-5.5的性能提升均在20-45个百分点，验证了策略的泛化能力。分析显示，界面修复、环境约定和操作知识的引入，是性能提升的关键因素。
Applications
该方法适用于企业IT运维、财务自动化、客户服务等场景，能显著提升任务完成效率和安全性。企业可通过外部调度策略快速适应环境变化，减少微调成本。未来，结合强化学习实现调度与模型的联合演化，将推动企业AI系统的自主学习和持续优化，助力企业数字化转型。
Limitations & Outlook
当前方法依赖于失败行为的准确划分，若环境变化频繁或失败模式模糊，效果可能受影响。演化过程计算成本较高，尤其在多模型迁移时需要多次验证。实际部署中，环境动态变化和安全风险未充分考虑，未来需结合在线学习和安全机制进行优化。
Plain Language Accessible to non-experts
想象你在厨房做饭，面对不同菜谱和食材，如何用同一套工具做出不同菜肴？传统方法就像每次都重新调试厨具和菜谱，而StarHarness像是提前准备好一套调味料和步骤，能根据不同菜肴自动调整。它通过观察失败的原因，优化调料和步骤，让厨具更聪明，做饭更快更好。这样，即使换菜谱，也不用重新调试，只需少量调整，就能做出满意的菜肴。这就像给厨房装上智能调度系统，让厨师（模型）在不同菜系中游刃有余，效率大大提升。
ELI14 Explained like you're 14
想象你在学校里玩一个游戏，你的目标是完成各种任务，比如解谜、答题或建造东西。刚开始，你用的工具（比如笔、尺子）都一样，但每次遇到不同的任务时，你会发现有些工具用得不好，或者需要调整。StarHarness就像是给你一套特别的工具箱，它会观察你做任务时遇到的问题，然后自动帮你调整工具，比如换个尺子、加个夹子，让你更快完成任务。它还会记住哪些调整效果最好，下次遇到类似问题时就能直接用。这样一来，你就不用每次都从头调工具，任务完成得更快、更顺利。这个方法让你的工具箱变得更聪明，帮你轻松应对各种挑战。
Glossary
分层搜索 (Stratified Search)
一种根据任务失败行为将搜索空间划分为不同层次的方法，以提高搜索效率。
用于任务调度策略的优化过程。
任务划分 (Task Partitioning)
将任务按照失败模式和性能指标划分为不同子集，便于有针对性地优化。
在演化过程中实现任务的系统化管理。
调度策略 (Scheduling Strategy)
控制任务执行顺序和资源分配的规则，用于提升整体效率。
在调度演化中核心优化目标。
模型迁移 (Model Transfer)
将经过调优的调度策略应用到不同模型或任务中，验证其泛化能力。
验证策略的实用性和普适性。
失败行为 (Failure Mode)
模型在任务中表现不佳的具体原因，如工具调用错误或环境不匹配。
任务划分和策略优化的基础。
Open Questions Unanswered questions from this research
- 1 如何在动态变化的企业环境中持续维护和更新调度策略？
- 2 多模型协同调度的有效机制尚未成熟，未来需要研究更高效的联合优化方法。
Applications
Immediate Applications
企业IT运维自动化
通过调度优化提升故障诊断和修复效率，减少人工干预，降低成本。
Long-term Vision
自主企业AI系统
实现模型与调度策略的联合自我优化，推动企业智能化全面升级，减少人力依赖。
Abstract
We present StarHarness, a framework for evolving environment-specific agent harnesses while keeping model weights fixed. The evolved harness can include prompt and task framing, tool interfaces, skills, MCP-backed providers, subagent structure, and agent-loop configuration. StarHarness constructs a compact evolution pool by stratifying tasks according to baseline failure behavior, separates proposer-visible search tasks from proposer-hidden selection tasks, and reserves held-out tasks for evaluating generalization. Across ITBench SRE, EnterpriseOps-Gym ITSM, and AutomationBench Finance, harness evolution improves full-benchmark performance by 20-35 percentage points over the default harness after 4-12 accepted changes per environment. These gains persist on tasks excluded from evolution and transfer without re-evolution across GPT and Qwen model families. Trace analysis links the improvements to interface repairs, environment conventions, and operational knowledge that compresses search, with fewer false-positive diagnoses and shorter trajectories in several settings. StarHarness therefore offers a practical way to reduce persistent model-environment mismatch in tool-rich enterprise tasks.