APTBench: Benchmarking Agentic Potential of Base LLMs During Pre-Training
APTBench evaluates base LLMs' agentic potential during pretraining via real-world trajectories, focusing on planning, action, and atomic skills.
Key Findings
Methodology
APTBench constructs evaluation questions from real-world successful trajectories, transforming complex multi-turn tasks into multiple-choice and text completion formats. It emphasizes core agentic skills—planning, action, atomic abilities—by extracting relevant content and generating distractors via model-based perturbations, validated by human experts. The framework supports long sequences, exceeding 16K tokens, to assess models’ long-context understanding and multi-turn decision-making. Experiments across models like Qwen3 series, Llama, DeepSeek show that agentic capabilities emerge at model sizes above 30B, with significant performance jumps indicating scale-dependent emergence phenomena.
Key Results
- Qwen3-1.7B scored an average of 24.27 on APTBench, while Qwen3-30B scored 41.60, demonstrating a clear scale effect. In software engineering, planning accuracy rose from 17.84% to nearly 50%, and action EM from 25.19% to 29.15%. Deep research tasks showed similar trends, with larger models outperforming smaller ones significantly.
- The results confirm that agentic abilities are scale-dependent, with a notable 'emergence' point around 30B parameters. The benchmark’s predictions correlate well with downstream agent performance, validating its utility.
- The framework’s low-cost, extensible design allows for broad application across domains, enabling early-stage assessment of models’ autonomous capabilities, guiding architecture and data strategies for better agentic performance.
Significance
This work addresses a major gap in evaluating models’ autonomous decision-making during pretraining, moving beyond static skill tests. It offers a practical, scalable method to predict and enhance models’ agentic abilities early in development, facilitating more capable AI systems for automation, research, and software engineering. By capturing long-term, multi-turn interactions, it aligns evaluation with real-world demands, reducing reliance on costly post-training fine-tuning and enabling more efficient model development cycles.
Technical Contribution
The paper introduces a novel trajectory-based task transformation framework, combining real-world data with negative sampling and human validation, to create long-sequence, multi-turn evaluation benchmarks. It innovates by focusing on core agentic skills within a pretraining context, revealing the non-linear emergence of capabilities at certain scales. The approach is highly extensible and compatible with various domains, providing a new paradigm for early prediction of autonomous abilities in large models.
Novelty
This is the first benchmark explicitly designed to evaluate the agentic potential of base models during pretraining, emphasizing multi-turn, long-context tasks derived from real trajectories. Unlike static skill tests, it captures dynamic decision-making and planning, offering a new perspective on model capabilities and scaling laws.
Limitations
- Current focus is limited to software engineering and deep research scenarios; applicability to other domains remains to be validated.
- Trajectory collection relies on manual validation, which may introduce bias or inconsistency.
- Model performance on small-scale models remains limited, requiring further optimization for resource-constrained environments.
Future Work
Future efforts will expand to additional domains such as autonomous agents and customer service, incorporate reinforcement learning for capability enhancement, and develop automated trajectory collection methods to reduce manual effort. Further research will explore the theoretical underpinnings of capability emergence and ways to lower the scale threshold for agentic abilities.
AI Executive Summary
The rapid development of large language models (LLMs) has revolutionized natural language processing, but assessing their true autonomous, agentic capabilities during pretraining remains a challenge. Traditional benchmarks focus on static skills like knowledge recall or reasoning, which do not reflect a model’s ability to plan, act, and adapt in dynamic environments. To bridge this gap, this paper introduces APTBench, a novel evaluation framework that leverages real-world successful trajectories to measure core agentic skills in base models.
APTBench transforms complex, multi-turn tasks from domains such as software engineering and deep research into multiple-choice and text completion questions. These questions are designed to evaluate the model’s planning, execution, and atomic abilities, with a focus on long sequences exceeding 16K tokens. The framework employs negative sampling and human validation to ensure question quality and difficulty. Extensive experiments across models of varying sizes reveal that agentic capabilities exhibit a non-linear emergence, with significant improvements observed beyond 30B parameters. This phenomenon underscores the importance of scale in developing autonomous AI systems.
The significance of this work lies in its ability to provide a low-cost, scalable, and early indicator of a model’s potential for autonomous task execution. By focusing on pretraining, it enables researchers and practitioners to guide model development more effectively, reducing reliance on costly post-training evaluations. The approach’s extensibility promises broad applicability across multiple domains, fostering the development of more capable, autonomous AI agents. While current results are promising, future work aims to expand the scope, incorporate reinforcement learning, and refine trajectory collection methods, ultimately advancing the frontier of autonomous AI capabilities.
Deep Analysis
Background
近年来，随着GPT、BERT、T5等预训练模型的崛起，模型在自然语言理解和生成任务中取得了巨大突破。传统评估指标如GLUE、MMLU主要关注静态知识和单一技能，难以反映模型在多轮交互、动态环境中的表现。随着模型在自动化任务中的应用需求增加，研究者开始关注模型的代理能力，包括规划、决策和执行能力。现有评估多依赖后训练微调或专门环境，成本高且难以在预训练阶段进行预测。如何在模型预训练早期有效预测其未来的自主能力，成为亟待解决的问题。
Core Problem
当前的评估体系无法反映模型在实际自主任务中的表现，主要因缺乏面向多轮交互、长上下文的早期指标。静态技能测试不能捕捉模型的规划、行动和反馈能力，导致实际应用中表现差异大。此外，代理评估多依赖昂贵的后训练微调，限制了预训练中能力的早期预测。缺少有效的早期指标，阻碍模型能力的优化和架构设计，亟需一种低成本、高效的评估工具。
Innovation
提出APTBench，创新点在于：
- �� 轨迹驱动的任务转化：采集真实场景中的成功轨迹，将复杂多轮任务转化为多项选择和文本补全题，适配基础模型。
- �� 长序列、多轮交互设计：利用长上下文，评估模型的规划、行动和原子能力，突破静态单轮测试的限制。
- �� 负样本生成与人工验证：确保题目难度和答案合理性，提高评估的准确性。
- �� 跨场景扩展能力：以软件工程和深度研究为例，未来可扩展至更多应用场景。
- �� 规模依赖分析：揭示模型规模与能力的“突变”现象，为模型设计提供理论支持。
Methodology
- �� 任务与轨迹采集：选择真实场景（如软件工程、深度研究），收集成功轨迹，确保轨迹的多轮交互和任务完成。
- �� 任务转化：提取轨迹中的规划、行动、原子能力内容，设计对应的多项选择题和文本补全题。
- �� 负样本生成：利用模型生成干扰选项，确保题目难度，提升评估的区分度。
- �� 人工验证：确保题目和答案的正确性，避免偏差。
- �� 长序列处理：设计支持超过16K tokens的输入，验证模型的长上下文理解能力。
- �� 规模分析：在不同模型规模上测试，观察代理能力的“突变”现象。
Experiments
在Qwen3系列（1.7B、4B、8B、30B）、Llama、DeepSeek等模型上进行评估，比较其在APTBench中的表现。指标包括规划准确率（ACC）、行动精确匹配（EM）和测试补全（ROUGE、EM）。实验涵盖不同模型规模、不同任务场景（软件工程、深度研究），并进行消融分析验证方法的有效性。结果显示，模型规模达到30B后，代理能力出现“突变”，表现出显著提升，验证了规模与能力的关系。
Results
模型规模达到30B后，APTBench得分出现“突变”，平均提升超过15点，验证规模与能力的关系。软件工程任务中，规划准确率从17.84%提升至49.93%，行动EM从25.19%到29.15%。深度研究场景中，模型表现出更强的推理和长序列处理能力，30B以上模型在任务完成度和推理准确性方面显著优于小模型。这些结果表明APTBench能有效捕捉模型的代理潜能，且规模是关键因素。
Applications
该基准可用于模型预训练阶段的能力筛选与优化，帮助设计更具自主性的模型架构。未来可结合强化学习技术，提升模型在复杂环境中的自主决策能力，广泛应用于自动化软件开发、科研辅助、智能助理等场景。其低成本、易扩展的特性，使得大规模预训练数据的生成与评估成为可能，推动行业自动化水平提升。
Limitations & Outlook
目前主要针对软件工程和深度研究场景，其他领域的适用性尚未验证。轨迹采集依赖人工验证，存在偏差。模型规模依赖明显，小模型表现有限，未来需优化低资源场景下的评估效果。
Plain Language Accessible to non-experts
想象你在一家大型厨房里做菜。每次做菜都需要按照食谱准备食材、调味、烹饪，厨师（模型）需要记住每一步，合理安排流程，还要根据厨师的经验调整步骤。传统的厨房评估只看厨师是否能记住食谱，但实际上，厨房里最重要的是厨师能根据不同情况灵活应变，合理安排每个步骤，最终做出美味的菜肴。APTBench就像是用真实厨房操作的录像，评估厨师在复杂多轮操作中的表现，帮助厨师变得更聪明、更灵活。
ELI14 Explained like you're 14
想象你在学校参加一个烹饪比赛。平时你只会记住菜谱，但比赛时你需要根据现场情况调整，比如多加点盐或者换个调料。这个比赛考验你的计划和行动能力，而不是只记住菜谱。APTBench就像是用真实的厨房录像，看看你能不能在比赛中灵活应对各种变化。它会把复杂的烹饪过程变成一系列简单的问题，比如下一步该做什么，或者怎么解决厨房里的突发问题。通过这个方法，评委可以知道你是不是一个真正会做菜的高手，不仅仅是记忆力强，更重要的是会应变和操作。
Glossary
Agentic Potential (代理潜能)
指模型在多轮交互中表现出的规划、行动和反馈能力，是衡量其自主完成复杂任务的潜力。
论文中用以描述模型在真实场景中自主执行任务的能力指标。
Trajectory (轨迹)
在任务执行中，模型或人类的连续操作和反馈过程，用于反映任务的完整执行路径。
用于构建APTBench中的任务题目，采集成功轨迹作为评估基础。
Long Context (长上下文)
模型处理的超长输入序列，超过常规模型的最大长度（如16K tokens），考察模型的长序列理解能力。
APTBench设计支持超过16K tokens的输入，评估模型的长序列处理能力。
Negative Sample (负样本)
通过扰动正确答案生成的干扰项，用于测试模型的辨别能力和鲁棒性。
在题目生成中用以确保题目的难度和区分度。
Scale Emergence (规模突变)
模型规模达到一定阈值后，代理能力出现显著提升的现象。
实验中发现30B模型后，代理能力有突变式增长。
Open Questions Unanswered questions from this research
- 1 如何在预训练中系统性增强模型的多轮交互能力，尤其是在低资源环境下的表现？
- 2 现有轨迹采集方法是否能全面覆盖不同任务场景的复杂交互？
- 3 如何进一步降低人工验证成本，提升自动化评估的准确性？
Applications
Immediate Applications
模型预训练能力筛选
在预训练阶段使用APTBench评估模型的代理潜能，指导模型架构和数据策略优化，提升后续微调效果。
多任务能力调研
通过不同场景的轨迹采集，分析模型在多任务、多轮交互中的表现差异，为模型设计提供数据支持。
Long-term Vision
自主任务执行系统
结合APTBench的评估结果，开发具有自主规划和执行能力的通用智能系统，应用于自动化科研、软件开发等。
Abstract
With the rapid development of LLM-based agents, there is a growing trend to incorporate agent-specific data into the pre-training stage of LLMs, aiming to better align LLMs with real-world autonomous task execution. However, current pre-training benchmarks primarily focus on isolated and static skills, e.g., common knowledge or mathematical/code reasoning, and fail to reflect model's agentic capabilities. On the other hand, agent benchmarks are typically designed for post-trained models, requiring multi-turn task execution abilities that base models struggle to support. Thus, there is a compelling need for a benchmark that can evaluate agentic potentials during pre-training and guide the model training more effectively. To address this gap, we propose APTBench, a framework that converts real-world agent tasks and successful trajectories into multiple-choice or text completion questions tailored for base models. It focuses on core agentic abilities, e.g., planning and action, and covers key agent scenarios, software engineering and deep research. Compared to existing general-purpose benchmarks, APTBench offers a more predictive signal of a model's downstream performance as an agent, while remaining significantly more lightweight and cost-effective than full-scale, end-to-end agent evaluations after post-training.