Glite ARF: Verifier-Driven Research with Parallel LLM Coding Agents
Verifier-driven framework enables parallel LLM agents with 29.9% RMSE reduction in BEA 2026, ensuring reproducibility and auditability.
Key Findings
Methodology
The framework employs a three-role architecture: human hypothesis proposer, coding agents (Claude Code, Codex CLI) executing tasks within fixed structures, and deterministic Python verifiers enforcing task isolation, immutability, and result integrity. Tasks are organized in folders with versioned specifications, verified before merging. The system supports multi-task parallelism via git branches and worktrees, with verification scripts detecting violations and applying corrections. This setup ensures high auditability, traceability, and minimal overhead (~1% wall-clock time). The approach was validated in multi-domain campaigns, notably BEA 2026, demonstrating significant performance gains and error detection capabilities.
Key Results
- In BEA 2026, the system achieved top rankings across all three languages (Spanish, German, Mandarin), reducing RMSE by an average of 29.9% in the closed track and 35.9% in the open track, outperforming the baseline. A total of 273 tasks (146 runs) were executed with up to 12 agents in parallel, costing approximately $450 in API usage, showcasing efficiency and scalability.
- Structural provenance enabled rapid identification and removal of leaked features, correcting an implausible RMSE of 0.609 to 0.802. The system's parallelism and verification mechanisms contributed to only about 1% increase in wall-clock time, confirming high efficiency.
- Across three campaigns in different domains, the framework demonstrated consistent structural robustness, supporting large-scale, multi-task autonomous research with reliable audit trails and minimal overhead.
Significance
This work addresses core challenges in large-scale autonomous scientific research: ensuring task isolation, data integrity, and result traceability amidst complex multi-agent workflows. By embedding rules into executable code, it reduces errors caused by low-level instruction lapses and manual oversight. The framework's success in competitive settings highlights its potential to transform AI-driven research, making it more reliable, auditable, and scalable, thus accelerating scientific discovery and industry applications.
Technical Contribution
The paper introduces a structured, verification-driven architecture combining task isolation, immutable artifacts, versioned specifications, and automated validation scripts. It leverages git-based parallelism and structured provenance to manage multi-agent workflows efficiently. This integration of engineering best practices with formal verification ensures high reliability and traceability, setting a new standard for automated research pipelines. The approach is compatible with existing multi-agent orchestration tools, extending their capabilities with rigorous structural guarantees.
Novelty
This is the first comprehensive implementation of a verifier-driven, rule-enforced research framework for multi-LLM agent campaigns. Unlike prior prompt-based or loosely structured systems, it embeds rules directly into code, ensuring strict compliance and auditability. Its combination of task isolation, versioned specifications, and automated verification represents a novel paradigm shift in AI research automation, addressing fundamental reliability issues that have limited prior efforts.
Limitations
- The system relies on human-driven hypothesis selection, limiting full automation. Fully autonomous hypothesis generation remains an open challenge.
- Verification scripts focus on structural correctness; semantic validation of scientific validity still requires human oversight, leaving room for subtle errors.
- Scaling to extremely large campaigns may encounter resource constraints and complex merge conflicts, necessitating further optimization.
Future Work
Future directions include integrating automated hypothesis generation via reinforcement learning, enhancing semantic verification through AI-based consistency checks, and optimizing resource scheduling for larger campaigns. Expanding multi-modal data support and developing more intelligent verification mechanisms will further improve robustness. These advancements aim to realize fully autonomous, trustworthy AI-driven scientific research pipelines.
AI Executive Summary
Automated scientific research using large language models (LLMs) has shown immense promise but faces critical challenges in ensuring reliability, reproducibility, and auditability. Traditional approaches relying on prompt engineering and loose orchestration often lead to errors, data leaks, and irreproducible artifacts, especially in large-scale multi-agent campaigns. To address these issues, this work introduces Glite ARF, a verification-driven framework that structures research workflows into a formal, rule-enforced system.
The core idea is to embed the rules of research—such as task isolation, artifact immutability, and provenance tracking—directly into executable code, using deterministic Python scripts as verifiers. Human researchers propose hypotheses, while coding agents execute tasks within isolated folders, with all outputs verified before being merged into the main project. This approach ensures that violations are detected immediately, preventing silent errors and facilitating audit trails.
The framework was validated in the BEA 2026 vocabulary difficulty prediction challenge, where it enabled the team to outperform baseline models significantly. Across 273 tasks involving multiple languages and feature sets, the system achieved an average RMSE reduction of nearly 30%, with the best models reaching a Pearson correlation of 0.91. The structural provenance allowed rapid identification and correction of data leaks, exemplifying the system’s robustness.
Beyond the specific application, Glite ARF demonstrates that rigorous, rule-based management of multi-agent research can dramatically improve reliability without sacrificing efficiency. Its parallel execution capability, supported by git-based task management, allows scaling to large campaigns with minimal overhead (~1%). The approach paves the way toward fully autonomous, trustworthy AI-driven scientific workflows, with ongoing work aimed at automating hypothesis generation and semantic validation. Despite current limitations, this framework represents a significant step toward reliable, auditable AI research at scale.
Deep Analysis
Background
近年来，自动化科研逐渐成为AI研究的重要方向。早期工作如Karpathy的autoresearch模式强调单指标优化，代表系统如Lu等的AI Scientist v1（2024）实现了端到端的论文生成，但在规模和可靠性方面仍存在瓶颈。多代理调度工具（AutoGen、MetaGPT）提供了协作基础，但缺乏严格的结构验证机制，导致任务违规、数据泄露和结果不可追溯的问题日益突出。随着模型复杂度提升，低效和不可靠的自动化流程限制了其应用范围。为解决这些问题，迫切需要引入结构化、验证驱动的管理框架，以确保研究的可信性和可追溯性。
Core Problem
当前自动化科研系统在多任务、多代理环境中容易出现数据泄露、任务违规和结果不可追溯的问题。这些问题源于缺乏严格的结构管理和验证机制，导致错误累积，影响研究的可靠性。尤其在多代理并行执行时，任务隔离和数据完整性难以保证，严重制约了自动化科研的规模和可信度。如何在保证效率的同时，实现任务的严格隔离、数据的不可变性和结果的可追溯，成为亟待解决的核心难题。
Innovation
本研究提出了验证驱动的多代理结构化框架，核心创新包括：
- �� 任务隔离：每个任务在独立文件夹和git工作树中执行，防止数据污染。
- �� 不可变性与修正覆盖：完成的任务文件夹不可修改，错误通过补丁修正，保证数据完整性。
- �� 结构化验证：每个输出文件有版本化规范，验证器自动检测违规行为。
- �� 多任务并行：利用git分支和工作树支持多代理同时运行，提升效率。
- �� 结构化追踪：源数据和中间结果全链路追溯，确保结果可审计。这些创新共同保障了大规模、多任务自动化研究的可靠性和效率。
Methodology
- �� 人类研究者提出假设和任务建议，存入建议库。
- �� 编码代理（Claude Code、Codex CLI）在固定结构下执行任务，任务存于独立文件夹和git分支。
- �� 任务文件夹内存放数据、模型和代码，验证器（verificator）在提交前检测违规行为。
- �� 任务执行过程中，代理输出符合版本规范的文件，验证器确保结构完整。
- �� 任务完成后，输出不可变，错误通过补丁修正，修正文件存于特定目录。
- �� 聚合器（aggregator）定期读取任务数据，应用修正，生成项目总览。
- �� 支持多任务并行，最多12个代理同时运行，资源由单机调度。
- �� 通过版本化规范和验证机制，确保每个步骤的可追溯性和审计能力。
Experiments
在BEA 2026竞赛中，团队使用该框架执行273个任务，涵盖129个特征集，涉及多语言（西班牙语、德语、普通话）。实验采用RMSE作为主要指标，比较官方基线，结果显示平均降低29.9%（封闭轨）和35.9%（开放轨）。模型包括LLaMA-3.1-8B LoRA，训练成本约450美元。多轮实验追踪模型性能变化，验证了框架的高效性和稳定性。通过结构化追踪，成功识别泄露目标变量的特征集，及时剔除，确保了结果的可信度。
Results
实验结果显示，利用Glite ARF架构，团队在多语言、多任务环境中实现了显著的性能提升，RMSE平均降低近30%，在竞赛中获得第一和第二名。结构化的源数据追踪和验证机制，有效识别并剔除泄露特征，确保了模型的公正性和可靠性。系统支持多代理并行，成本低廉，效率高，验证了其在大规模自动化科研中的实用性。多轮实验中，系统仅增加约1%的墙钟时间，展现出优异的扩展性和稳定性。
Applications
该框架适用于大规模自动化科研、跨语言模型调优、特征工程和模型验证等场景。科研机构和企业可以利用其结构化管理和验证机制，提升实验的可靠性和审计能力。未来，结合自动假设生成和强化学习，有望实现更自主的科研流程，推动AI在科学探索中的深度应用。
Limitations & Outlook
目前系统仍依赖人工在假设选择上的引导，自动化程度有限。验证器主要保证结构和数据完整性，语义合理性仍需人工判断。硬件成本和调度复杂性在大规模部署中仍是挑战，未来需优化资源调度和验证机制以实现更全面的自主性。
Plain Language Accessible to non-experts
想象你在一个大型工厂里，每个工序都由不同的工人负责。为了确保每个工序都按标准完成，工厂制定了严格的规则，比如每个工人只能在自己的区域工作，完成后不能随意改动之前的工作，还要有专门的检查员来验证每个部分是否符合标准。这样，即使有很多工人同时工作，也能保证整个生产线的质量和追溯性。Glite ARF就像这个工厂的管理系统，把研究中的每个任务当成一个工序，利用规则和验证机制，确保每一步都可追溯、不可篡改，从而让自动化研究变得可靠又高效。
ELI14 Explained like you're 14
想象你在学校里做一个大项目，你需要很多不同的小组合作。每个小组负责不同的部分，比如写报告、做实验、画图。为了确保每个小组的工作都能被检查和追踪，你会制定一些规则，比如每个小组的工作要存放在自己专属的文件夹里，完成后不能随意改动之前的内容，还要有老师来检查每个部分是否符合要求。这样，不管有多少小组同时工作，老师都能清楚每个部分的来龙去脉，确保整个项目的质量。Glite ARF就像这个老师的系统，用规则和自动检查，保证自动化研究的每一步都可靠、可追溯，避免出错。
Abstract
LLM coding agents make it tempting to automate empirical research by delegating experiments to them directly, but naive delegation does not scale to large projects: low-rate instruction lapses compound into broken, irreproducible artefacts. To address this problem, we present Glite ARF, an open-source Python framework for running many LLM coding agents in parallel on a research repository without sacrificing reproducibility or auditability. The framework defines a three-role stack: a human researcher chooses which hypotheses to test, coding agents (Claude Code, Codex CLI) implement individual tasks under a fixed structure, and deterministic Python verifier scripts enforce task isolation, immutability of completed work, a corrections overlay, and a materialised project overview. We call this verifier-driven research: the rules of the research process live in code that fails loudly when violated, not in prose that agents are merely asked to follow. Using Glite ARF, we developed our submission to the BEA 2026 vocabulary-difficulty shared task, placing first in the closed track and second in the open track on all three target languages (Spanish, German, Mandarin) and reducing the official baseline RMSE by 29.9% (closed) and 35.9% (open). The campaign comprised 273 tracked tasks (146 experiment runs) across 129 feature sets, run by up to twelve parallel agents orchestrated from a single laptop - with some model training on rented A100s - at approximately \$450 in LLM API spend (\$498 total third-party cost), and structured per-fold provenance let us catch and strip four target-leaking feature sets, correcting an implausible 0.609 RMSE to 0.802. Across three campaigns in three domains, the framework's structural machinery adds only about 1% of wall-clock time. Framework and a public demo project accompany this paper.