A-MapReduce: Executing Wide Search via Agentic MapReduce
A-MapReduce optimizes wide search tasks using a MapReduce-inspired framework, achieving 5.11%-17.50% Item F1 improvements.
Key Findings
Methodology
A-MapReduce redefines wide search as a horizontally structured retrieval problem using a MapReduce-inspired framework. It employs task decomposition, parallel execution, and result aggregation, coupled with experiential memory for adaptive optimization.
Key Results
- On WideSearch, A-MapReduce improves Item F1 by 12.71% and Row F1 by 13.02%, with a 4.23% increase in success rate.
- On DeepWideSearch, core entity accuracy improves by 3.52%-19.09%, and Column F1 increases by 9.71%-30.18%.
- Reduces runtime by 45.8% compared to baselines, saving $1.10 per task on average.
Significance
This work significantly enhances efficiency and accuracy in wide search tasks, addressing bottlenecks in long-horizon execution and large-scale retrieval, with broad academic and industrial implications.
Technical Contribution
Introduces a MapReduce-inspired multi-agent framework, modeling wide search as a horizontal retrieval problem. It enables cross-task optimization through experiential memory, offering new theoretical and engineering possibilities.
Novelty
A-MapReduce is the first to apply the MapReduce paradigm to multi-agent systems, offering a novel horizontal retrieval perspective distinct from traditional vertical reasoning frameworks.
Limitations
- Experiential memory may introduce overhead in extremely large-scale tasks.
- Framework relies on high-performance LLMs, requiring significant computational resources.
- Adaptability to complex queries remains partially validated.
Future Work
Future research could explore memory compression, broader task applicability, and optimization for low-resource environments.
AI Executive Summary
Existing multi-agent systems struggle with wide search tasks due to their reliance on vertical recursive reasoning, which is inefficient for large-scale horizontal retrieval. A-MapReduce addresses this by redefining wide search as a horizontally structured problem, leveraging task decomposition, parallel execution, and result aggregation.
The framework incorporates experiential memory to extract execution patterns from past tasks, enabling continuous optimization. On benchmarks like WideSearch and DeepWideSearch, A-MapReduce achieves up to 17.50% Item F1 improvement and reduces runtime by 45.8%.
While challenges like storage overhead and query adaptability remain, A-MapReduce offers a transformative approach to wide search tasks, with significant potential for academic and industrial applications. Future work will focus on memory efficiency and broader applicability.
Deep Analysis
Background
Multi-agent systems have excelled in areas like reasoning and retrieval but are primarily designed for vertical recursive reasoning, limiting their efficiency in wide search tasks.
Core Problem
Wide search requires large-scale horizontal retrieval and aggregation. Existing systems face bottlenecks in target management and execution efficiency, making them unsuitable for such tasks.
Innovation
A-MapReduce introduces a MapReduce-inspired framework for wide search. Key innovations include task decomposition, parallel execution, result aggregation, and experiential memory for adaptive optimization.
Methodology
- �� Task decomposition: Splits retrieval tasks into atomic units.
- �� Parallel execution: Distributes tasks across agents for simultaneous processing.
- �� Result aggregation: Combines partial outputs into a unified result.
- �� Experiential memory: Records execution patterns to refine future task strategies.
Experiments
Experiments were conducted on WideSearch and DeepWideSearch benchmarks, comparing against diverse baselines. Metrics included Item F1, Row F1, and runtime.
Results
A-MapReduce improves Item F1 by 12.71% on WideSearch and core entity accuracy by 19.09% on DeepWideSearch. Runtime is reduced by 45.8%.
Applications
Applicable to large-scale information retrieval, knowledge graph construction, and data mining, especially in scenarios requiring efficient horizontal retrieval.
Limitations & Outlook
High storage overhead, limited query adaptability, and dependency on high-performance LLMs.
Plain Language Accessible to non-experts
Imagine A-MapReduce as a smart factory. Each wide search task is like a big order, which the factory breaks into smaller tasks. Workers (agents) handle these tasks in parallel. The factory then combines the results into a final product. With experiential memory, the factory learns to optimize its processes over time, becoming faster and more efficient.
ELI14 Explained like you're 14
Think of playing a treasure hunt game with a huge map. A-MapReduce is like a clever team leader who divides the map into sections and assigns each teammate a part. After everyone finds treasures, the leader combines all the findings. Even cooler, the leader remembers the best strategies for future hunts, making the team faster each time!
Glossary
MapReduce
A distributed computing framework that splits tasks into mapping and reducing stages.
Forms the core design of A-MapReduce.
Wide Search
A task emphasizing large-scale horizontal retrieval.
The primary application of A-MapReduce.
Experiential Memory
A mechanism to store execution patterns from past tasks.
Used to optimize task allocation and execution strategies.
Item F1
A metric combining precision and recall to evaluate retrieval accuracy.
Used to assess model performance in experiments.
Parallel Execution
Executing multiple tasks simultaneously to improve efficiency.
A key feature of A-MapReduce.
Open Questions Unanswered questions from this research
- 1 How can experiential memory storage overhead be minimized?
- 2 How can the framework be optimized for low-resource environments?
Applications
Immediate Applications
Knowledge Graph Construction
Efficiently builds large-scale knowledge graphs through optimized retrieval and aggregation.
Data Mining
Extracts and analyzes information from large datasets efficiently.
Long-term Vision
Generalized Intelligent Search Engines
Develops next-gen search systems capable of handling complex wide search tasks.
Abstract
Contemporary large language model (LLM)-based multi-agent systems exhibit systematic advantages in deep research tasks, which emphasize iterative, vertically structured information seeking. However, when confronted with wide search tasks characterized by large-scale, breadth-oriented retrieval, existing agentic frameworks, primarily designed around sequential, vertically structured reasoning, remain stuck in expansive search objectives and inefficient long-horizon execution. To bridge this gap, we propose A-MapReduce, a MapReduce paradigm-inspired multi-agent execution framework that recasts wide search as a horizontally structured retrieval problem. Concretely, A-MapReduce implements parallel processing of massive retrieval targets through task-adaptive decomposition and structured result aggregation. Meanwhile, it leverages experiential memory to drive the continual evolution of query-conditioned task allocation and recomposition, enabling progressive improvement in large-scale wide-search regimes. Extensive experiments on five agentic benchmarks demonstrate that A-MapReduce is (i) high-performing, achieving state-of-the-art performance on WideSearch and DeepWideSearch, and delivering 5.11% - 17.50% average Item F1 improvements compared with strong baselines with OpenAI o3 or Gemini 2.5 Pro backbones; (ii) cost-effective and efficient, delivering superior cost-performance trade-offs and reducing running time by 45.8\% compared to representative multi-agent baselines. The code is available at https://github.com/mingju-c/AMapReduce.