On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents
This study empirically shows that AGENTS.md files reduce AI coding agents' token usage by 20% and task completion time by over 20% on GitHub pull requests.
Key Findings
Methodology
Using an empirical approach, 10 open-source repositories with 124 PRs were analyzed. The study compared agent performance with and without AGENTS.md files, controlling for task, repository state, and model version (GPT-5.2-Codex). Metrics included wall-clock time and token consumption, measured under paired conditions. Content filtering ensured relevance of AGENTS.md files, focusing on project description, architecture, and conventions. Experiments were run in isolated Docker environments, with random sampling for output validation. Statistical significance was assessed via Wilcoxon signed-rank tests, confirming the observed efficiency gains.
Key Results
- Presence of AGENTS.md reduced average token consumption from 5744.81 to 4591.46 (~20%), and median tokens from 2925 to 2440 (~16.58%). Wall-clock time decreased from 162.94s to 129.91s (~20%), with median time dropping from 98.57s to 70.34s (~28.64%).
- Efficiency improvements were statistically significant (p<0.05), indicating that AGENTS.md files guide agents to perform tasks faster and with fewer resources.
- Content analysis suggests that project structure and conventions described in AGENTS.md facilitate quicker navigation and reasoning, leading to resource savings.
Significance
This research provides the first quantitative evidence that repository-level instruction files like AGENTS.md significantly enhance AI agent efficiency in real-world development tasks. It addresses the long-standing challenge of optimizing autonomous code generation workflows, demonstrating that well-maintained configuration files can reduce computational costs and improve throughput. These findings have implications for both academia and industry, promoting better integration of AI agents into continuous development pipelines and encouraging standardized documentation practices.
Technical Contribution
The study introduces a rigorous evaluation framework combining paired experimental design, content filtering, and statistical validation to quantify the impact of AGENTS.md files. It highlights the importance of repository-specific instructions in guiding AI behavior, providing a methodological template for future efficiency studies. The work bridges the gap between configuration management and AI performance, offering a pathway for systematic optimization of autonomous development agents.
Novelty
This is the first empirical study to isolate and measure the effect of repository-level AGENTS.md files on AI coding agent efficiency. Unlike prior work focusing solely on prompt engineering or model performance, this research emphasizes persistent configuration artifacts as a key factor influencing resource consumption and task speed, marking a novel direction in AI-assisted software engineering.
Limitations
- Limited to 10 repositories and 124 PRs, future work should include larger, more diverse datasets to validate generality.
- Only GPT-5.2-Codex was tested; other models and architectures may respond differently to AGENTS.md files.
- Metrics focus on resource usage, lacking comprehensive assessment of output correctness, maintainability, or functional quality.
Future Work
Future research will expand dataset size, include multiple models, and analyze the influence of content richness and structure in AGENTS.md files. Integrating correctness and maintainability metrics will provide a holistic view of impact. Developing automated tools for generating and optimizing configuration files could further enhance efficiency. Long-term, standardizing repository configuration practices could transform AI-assisted development workflows.
AI Executive Summary
The rapid evolution of AI coding agents like OpenAI Codex has transformed software development, enabling autonomous code generation, testing, and review. However, optimizing their performance remains a challenge. This study investigates the role of repository-level instruction files, specifically AGENTS.md, in enhancing agent efficiency during real-world tasks. By analyzing 124 pull requests across 10 open-source repositories, the researchers compared agent performance with and without AGENTS.md files, controlling for task complexity and repository state.
Results demonstrate that the presence of AGENTS.md significantly reduces resource consumption: token usage drops by approximately 20%, and task completion time decreases by over 20%. These improvements are statistically significant and consistent across different repositories, indicating that detailed, project-specific instructions help agents navigate and reason more effectively. Content analysis reveals that instructions related to project structure and conventions are particularly influential.
This research provides compelling empirical evidence that well-maintained configuration files can serve as practical levers for optimizing AI-assisted development workflows. It underscores the importance of repository documentation in guiding autonomous agents, paving the way for standardized practices that can lower costs and accelerate software delivery.
Looking ahead, expanding the dataset, testing diverse models, and integrating quality assessments will deepen understanding of how configuration influences not only efficiency but also correctness and maintainability. The findings suggest that investing in structured, persistent instructions like AGENTS.md can unlock significant gains in automation and productivity, ultimately transforming the landscape of AI-driven software engineering.
Deep Analysis
Background
随着AI编码代理如OpenAI Codex和Claude在软件开发中的应用日益普及，研究重点逐渐从模型性能转向如何提升自动化效率。早期工作主要关注模型的准确性和任务成功率，缺乏对仓库配置文件影响的系统性分析。近年来，AGENTS.md等仓库级指令文件被提出，用于描述项目结构、编码规范和操作流程，旨在提升代理的行为一致性和效率。已有研究显示，这些文件在功能指导中发挥重要作用，但缺乏实证数据验证其在实际开发中的效果。本研究填补了这一空白，首次量化AGENTS.md对代理运行时间和Token消耗的影响，为优化自动化开发流程提供理论依据。
Core Problem
尽管仓库配置文件逐渐成为行业标准，但其对AI编码代理性能的具体影响尚未被充分验证。开发者面临的核心问题是如何设计和维护高效的配置文件，以降低资源消耗、提升任务完成速度。缺乏系统性实证研究使得配置优化缺乏科学依据，限制了其在工业界的推广。解决这一问题需要深入分析不同内容配置对代理行为的影响机制，明确哪些内容能带来最大效率提升，从而指导实践中的配置策略。
Innovation
本研究的创新点包括：1）引入内容筛选机制，确保AGENTS.md内容与项目结构和规范高度相关；2）采用Paired设计，控制任务和仓库状态，确保结果的可靠性；3）结合Token和时间指标，量化配置文件对效率的影响。这些创新使得研究结果具有较强的实证性和指导性，为仓库配置优化提供了科学依据。不同于传统模型优化方法，本研究强调配置文件的持久性和内容结构，推动了配置管理在智能开发中的应用创新。
Methodology
- �� 选择GPT-5.2-Codex模型，确保模型版本一致。• 从已有仓库样本中筛选符合条件的仓库，确保每个仓库仅含一个根目录AGENTS.md文件。• 利用内容分类体系筛选内容，确保文件涵盖项目描述、架构和规范。• 从每个仓库随机抽取符合条件的已合并PR，确保PR规模小（≤100行变动，≤5文件）。• 在仓库历史状态下，模拟PR前状态，提取对应AGENTS.md内容。• 生成标准化任务描述，确保输入一致性。• 在Docker环境中，分别运行有无AGENTS.md的两组实验，控制变量，测量Token和时间。• 采样输出，验证任务完成情况，确保效率指标反映真实表现。
Experiments
实验在真实GitHub仓库中进行，采用随机抽样的10个仓库，每个仓库15个PR。每个PR在相同仓库状态下，分别运行有无AGENTS.md条件。指标包括Token总数和墙钟时间，统计学检验确保差异显著。通过内容筛选确保AGENTS.md内容相关性，控制任务复杂度，避免大规模变更干扰。多次重复实验，确保数据的稳定性和可靠性。实验还包括输出质量的随机抽样验证，确保效率提升不是由任务失败或无效输出驱动。
Results
实验结果显示，AGENTS.md文件的存在显著降低Token消耗和时间成本。平均Token使用从5744.81降至4591.46，减少约20%；墙钟时间由162.94秒降至129.91秒，减少约20%。中位数变化也支持这一结论，说明效率提升具有普遍性。这些数据验证了配置文件在引导代理行为、减少探索和推理时间方面的作用，为自动化开发提供了实证依据。
Applications
该研究结果可指导开源项目和企业在配置AI编码代理时，优先设计和维护AGENTS.md文件，以提升自动化效率。适用场景包括持续集成、自动修复和代码审查等。未来，结合内容丰富度和结构优化，将进一步推动智能开发工具的标准化和工业化应用，降低开发成本，加快软件交付速度。
Limitations & Outlook
本研究样本有限，未来需扩大规模验证普适性。模型仅使用Codex，其他模型表现尚未验证。效率指标主要集中在Token和时间，未充分评估输出正确性和维护性。内容筛选依赖人工和模型分类，可能存在偏差。未来应结合多模型、多任务、多指标的综合评估，完善配置文件设计指南。
Plain Language Accessible to non-experts
想象一个工厂里生产不同的产品。工厂有一份详细的操作手册（就像AGENTS.md文件），告诉工人们如何组装、检验和包装产品。有了这份手册，工人们可以更快、更准确地完成工作，不用每次都自己摸索流程。没有手册时，工人可能需要花更多时间试错，甚至做错，导致生产变慢，浪费资源。这个研究发现，给AI编码代理提供类似的“操作手册”能让它们更高效地完成任务，减少不必要的计算和等待时间，就像工厂里有了详细的操作指南一样。这样，软件开发就能变得更快、更省钱，也更可靠。
ELI14 Explained like you're 14
想象你在厨房里做饭。平时你可能会随意放调料，试试味道，但这样可能会浪费时间，也做不好菜。有时候，厨师会用一本菜谱，告诉你用什么材料、怎么做。这个菜谱就像论文中的AGENTS.md文件，给AI提供了详细的做饭指南。有了菜谱，你可以更快做出好菜，不会浪费食材，也不用反复试错。研究发现，给AI“菜谱”后，它们在做饭（写代码）时会用更少的调料（Token）和花更少的时间（墙钟时间），效率大大提高。就像有了好菜谱，厨房变得更顺畅，做饭也变得更轻松。
Abstract
AI coding agents such as Codex and Claude Code are increasingly used to autonomously contribute to software repositories. However, little is known about how repository-level configuration artifacts affect operational efficiency of the agents. In this paper, we study the impact of AGENTS$.$md files on the runtime and token consumption of AI coding agents operating on GitHub pull requests. We analyze 10 repositories and 124 pull requests, executing agents under two conditions: with and without an AGENTS$.$md file. We measure wall-clock execution time and token usage during agent execution. Our results show that the presence of AGENTS$.$md is associated with a lower median runtime ($Δ28.64$%) and reduced output token consumption ($Δ16.58$%), while maintaining a comparable task completion behavior. Based on these results, we discuss immediate implications for the configuration and deployment of AI coding agents in practice, and outline a broader research agenda on the role of repository-level instructions in shaping the behavior, efficiency, and integration of AI coding agents in software development workflows.