AgentSearchBench: A Benchmark for AI Agent Search in the Wild

With the rapid advancement of artificial intelligence technology, AI agent systems are increasingly being applied across various fields. These agents can not only reason and plan but also interact with external tools and services to complete multi-step objectives. The progress of modern agent systems has led to a rapidly expanding ecosystem of agentic components, ranging from general-purpose assistants to highly specialized task-oriented modules. As humans increasingly rely on agents developed by diverse third-party providers, a fundamental challenge arises: how to select suitable agents for a given task. Traditional tools are typically scoped to specific operations, while agent capabilities are often more compositional and execution-dependent, making them difficult to assess without observing task outcomes. Textual descriptions provide only a partial signal of real competence, as agents with similar descriptions may perform differently in practice, while semantically dissimilar agents can achieve comparable results. This semantic-performance misalignment is further amplified in large and open agent ecosystems, where overlapping functionalities and non-uniform description formats make capability comparison difficult. Consequently, agent search is fundamentally more complex than conventional tool retrieval or model selection.

The core problem of agent search lies in how to retrieve and rank suitable agents from a large candidate repository given a user task. Traditional information retrieval typically determines relevance through static content matching, whereas agent search requires assessing functional capability through task execution. Agent search operates under different levels of task specification, including executable task queries and high-level task descriptions. Executable task queries are concrete and can be directly evaluated by running an agent, while high-level task descriptions are not directly executable. To evaluate agent capability under these settings, each task description is associated with a set of executable task queries, which instantiate the high-level goal under different concrete scenarios. Agent relevance is then determined based on consistent performance across these task instances, rather than relying on textual similarity or single-task outcomes.

The core innovations of AgentSearchBench include:

1. Formalizing agent search as an execution-dependent retrieval and reranking problem, highlighting the gap between semantic similarity and actual performance.

2. Constructing a large-scale benchmark with nearly 10,000 real-world agents, supporting both executable task queries and high-level task descriptions, and defining relevance through execution signals.

3. Providing a scalable evaluation pipeline that generates task instances and converts execution outcomes into fine-grained relevance annotations for retrieval and ranking assessment.

4. Demonstrating that lightweight behavioral signals, including execution-aware probing, can significantly improve ranking quality, emphasizing the importance of incorporating execution signals into agent discovery.

• �� Constructing AgentSearchBench: Collecting nearly 10,000 real-world agents from multiple providers, forming a large-scale agent repository.
• �� Task Query Construction: Synthesizing executable task queries from agent documentation using document-grounded task generation methods.
• �� Relevance Annotation: Generating fine-grained relevance annotations through execution signals, evaluated using a 5-point LLM-as-judge.
• �� Task Description Construction: Constructing task descriptions by abstracting high-level objectives from clusters of semantically related queries.
• �� Retrieval and Reranking Evaluation: Using execution signals for retrieval and reranking evaluation, reporting precision, recall, NDCG, and completeness.

The experimental design includes extensive benchmarking using AgentSearchBench to evaluate the performance of different retrieval and reranking methods on both executable task queries and high-level task descriptions. Baselines used include sparse, dense, tool-aware, and decoder embedding models. Experiments evaluate an average of 20 agents per query, with a total of 66,740 executions. Evaluation metrics include precision, recall, NDCG, and completeness. Experiments also explore the impact of lightweight behavioral signals on ranking quality.

Experimental results show that on task queries, tool-aware retrievers outperform sparse and dense baselines, while on task descriptions, dense retrievers become more competitive, with BGE achieving the strongest overall performance. However, performance drops significantly when moving from executable queries to high-level task descriptions, and completeness remains low across all methods, highlighting the difficulty of retrieving agents that can fully satisfy abstract requirements. Results indicate that while retrieval can capture coarse relevance, it struggles to identify agents with comprehensive task-solving capability, especially under high-level task specifications without explicit executable demands.

Application scenarios for AgentSearchBench include:

1. Conducting agent search in open ecosystems, supporting both executable task queries and high-level task descriptions.

2. Providing a large-scale benchmark for academic research to evaluate the performance of different retrieval and reranking methods.

3. Offering a tool for industry to select suitable agents in complex task environments, especially when dealing with abstract and multi-step tasks.

Limitations of AgentSearchBench include:

1. A significant performance drop when handling high-level task descriptions, indicating the difficulty of retrieving agents without explicit executable demands.

2. Existing retrieval and reranking methods remain limited in capturing execution-dependent capabilities, especially for abstract and multi-step tasks.

3. Applying AgentSearchBench in larger-scale and more complex task environments remains a challenge, and future research needs to explore how to improve agent search ranking quality in these environments.