Anchor: Mitigating Artifact Drift in Agent Benchmark Generation
Anchor formalizes enterprise workflows via constraint programming, mitigating artifact drift and enabling controlled task generation with certified solutions.
Key Findings
Methodology
The proposed Anchor pipeline formalizes business processes into parametric constraint satisfaction problems (CP-SAT). It generates instruction, environment setup, ground-truth solution, and verifier from a single solved specification, ensuring artifact consistency. By adjusting parameters, it controls task difficulty and produces a diverse, verifiable long-tail dataset, ERP-Bench. This approach combines formal modeling with automated artifact generation, significantly reducing artifact drift and enhancing environment auditability, thus providing a reliable benchmark for enterprise AI evaluation.
Key Results
- ERP-Bench contains 300 tasks covering procurement and manufacturing workflows, with certified optimal solutions. Models satisfy explicit constraints in 26.1% of trials but reach optimality in 17.4%. Across 18,000 trials, pass@5 scores decline monotonically from 70.5% (easy) to 22.3% (hard) in coding environments, demonstrating effective difficulty prediction via parameters.
- Parameter settings strongly correlate with realized difficulty, and model performance drops as task complexity increases. The results reveal a gap between constraint satisfaction and full optimality, highlighting current models' limitations in strictly adhering to business rules.
- The methodology ensures all four artifacts are generated deterministically from the same specification, effectively mitigating artifact drift. This leads to more reliable, auditable environments for enterprise AI training and evaluation.
Significance
This work addresses the critical challenge of artifact drift in enterprise task benchmarking by introducing a formalized, automated generation pipeline. It enhances the realism, verifiability, and controllability of complex business scenarios, facilitating more trustworthy AI model training and assessment. The creation of ERP-Bench offers a standardized, scalable platform for evaluating AI agents on real-world enterprise workflows, supporting both research and industry needs for robust, auditable AI systems. The approach paves the way for systematic, formalized enterprise AI benchmarking, promoting transparency and reproducibility in AI evaluation.
Technical Contribution
The core innovation lies in translating business workflows into formal constraint satisfaction problems solved by CP-SAT, enabling deterministic, consistent artifact generation. This contrasts with prior noisy or handcrafted datasets, providing a principled, scalable approach to task creation. The parameterization allows fine-grained control over task difficulty, supporting curriculum learning. The pipeline integrates expert knowledge with automated solving and artifact compilation, ensuring consistency and verifiability, thus advancing the state-of-the-art in enterprise AI benchmarking.
Novelty
This is the first work to systematically formalize enterprise workflow tasks as parametric constraint satisfaction problems for automated, consistent generation of instruction, environment, solution, and verifier artifacts. Unlike previous synthetic or manual datasets, Anchor guarantees artifact alignment by construction, effectively solving the artifact drift problem. Its ability to control task difficulty via parameters and produce certified solutions distinguishes it from existing benchmarks, offering a new paradigm for enterprise AI evaluation.
Limitations
- The approach depends heavily on expert-designed constraint models; inaccuracies or incompleteness in these models can lead to invalid tasks. Updating rules dynamically is non-trivial.
- Current methods focus on structured, formalizable workflows; unstructured or tacit knowledge-based tasks remain challenging.
- Computational costs of solving large constraint problems may limit scalability for more complex or larger-scale scenarios.
Future Work
Future directions include expanding to other enterprise domains such as sales, HR, and healthcare workflows, integrating natural language understanding for more flexible task descriptions, and improving solver efficiency for larger problems. Additionally, incorporating dynamic rule updates and learning-based constraint refinement could enhance adaptability. Developing more comprehensive validation and robustness checks will further improve task quality, supporting broader deployment in industry for AI training, benchmarking, and automation.
AI Executive Summary
In the rapidly evolving landscape of enterprise automation, creating reliable and scalable benchmarks for AI agents remains a significant challenge. Traditional methods often suffer from artifact drift, where the instruction, environment, solution, and verification components become misaligned, undermining the validity of evaluation environments. This disconnect hampers the development of trustworthy AI systems capable of handling complex, long-horizon business tasks.
To address this, the authors introduce Anchor, a formalized task-generation pipeline that leverages constraint satisfaction programming (specifically, Google OR-Tools CP-SAT) to translate business workflows into a unified, parametric specification. This specification serves as a single source of truth, from which all four critical artifacts—natural language instructions, environment configurations, solver-certified ground-truth solutions, and state-based verifiers—are deterministically derived. By doing so, Anchor ensures artifact consistency, significantly reducing the artifact drift problem.
Building on this foundation, the authors develop ERP-Bench, a benchmark comprising 300 long-horizon tasks modeled after real-world procurement and manufacturing workflows within the Odoo ERP system. These tasks span various complexity levels, with parameters controlling difficulty and certifiable optimal solutions. Extensive experiments with five state-of-the-art models across multiple environments demonstrate that task difficulty correlates strongly with model performance, with pass@5 scores decreasing from 70.5% on easy tasks to 22.3% on hard ones. Despite high constraint satisfaction rates, the models often fall short of achieving full optimality, revealing current limitations in business rule adherence.
Overall, this work provides a robust, auditable framework for enterprise AI benchmarking, enabling more trustworthy evaluation and training of AI agents in complex business scenarios. It opens avenues for broader application, including automation of diverse workflows and integration with natural language processing to further enhance scalability and realism. While challenges remain—such as model scalability and handling unstructured data—the proposed approach marks a significant step toward systematic, formalized enterprise AI evaluation, with promising implications for both academia and industry.
Deep Dive
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂里有很多不同的机器和工人，每个人都按照一定的规则做事。有时候，工厂需要生产不同的产品，但每次流程都不一样。以前，工人自己想办法安排流程，结果每次都不一样，有时还会出错。现在，有个智能系统像个聪明的老板，它用一套数学规则把所有流程都写下来，确保每次生产都按照这些规则进行。这个系统还能根据任务的难度调整流程的复杂程度，让新手和老手都能用。这样，工厂的生产变得更可靠、更高效，也方便检查每一步是否正确。就像用数学公式保证工厂每次都按规矩做事，没有偏差一样。
ELI14 Explained like you're 14
想象你在学校的厨房里做饭。以前，你可能会随意加料，结果每次做出来的味道都不一样。有时候还会忘记放盐或者放太多。现在，假设有个智能厨师，它会告诉你每一步该放什么、放多少，确保每次做的菜都一样好吃。这个厨师用一套数学规则，把所有步骤都写下来，然后根据你想做的菜难度调节步骤的复杂程度。比如，简单的菜只需要几步，复杂的菜需要更多步骤。这样，不管你是新手还是高手，都能做出好菜，而且每次都一样。这就像用数学公式保证厨房的每次操作都正确，没有偏差一样。
Abstract
AI agents are beginning to complete valuable, long-horizon business operations tasks, but training and evaluation environments for enterprise work still struggle to balance realism, verifiability, and scale. Environment and task creation frequently suffers from a failure mode we call artifact drift: when instructions, environments, oracles, and verifiers are created by loosely coupled processes, they frequently disagree on what a task requires, producing environments that are unsolvable, reward-hackable, or inconsistent. We introduce Anchor, a task-generation pipeline that formalizes domain experts' specifications of business workflows into constraint optimization programs. From a single parametric specification, the pipeline jointly produces a natural-language instruction, environment configuration, solver-certified ground-truth solution, and state-based verifier. With Anchor, altering parameters yields new tasks with controlled difficulty and known optimal solutions, producing harness-agnostic environments whose rewards depend solely on end-state business correctness. We apply Anchor to produce ERP-Bench: a benchmark of 300 long-horizon tasks spanning procurement and manufacturing workflows in a production-grade ERP system. We find that generation parameters predict realized difficulty, and that frontier models satisfy explicit task constraints in 26.1% of trials but reach a fully optimal solution in only 17.4% of trials. Overall, we show that Anchor and ERP-Bench offer a concrete recipe for building auditable evaluation environments for economically valuable agent work. We release the task generator and ERP-Bench dataset at erpbench.ai