SPAR: Scholar Paper Retrieval with LLM-based Agents for Enhanced Academic Search
SPAR employs multi-agent architecture with RefChain and query evolution, achieving up to +56% F1 on AutoScholar.
Key Findings
Methodology
SPAR comprises five agents: Query Understanding, Retrieval, Query Evolver, Judgment, and Reranker. It utilizes RefChain for citation expansion, multi-source retrieval, and iterative query refinement. The system leverages LLMs like Qwen3-32B for intent detection, keyword extraction, and relevance scoring. The framework is training-free, modular, and integrates expert-annotated datasets to build SPARBench, a high-quality benchmark. Experiments demonstrate F1 improvements of 56% on AutoScholar and 23% on SPARBench, outperforming baselines including traditional engines and prior agent methods.
Key Results
- On AutoScholar, SPAR achieved an F1 of 0.3843, surpassing PaSa (0.2449) by 56%, with balanced recall (0.4105) and precision (0.3612).
- On SPARBench, F1 reached 0.3015, outperforming all baselines, with recall 0.3103 and precision 0.2932.
- Query understanding modules improved precision from 0.21 to 0.34, demonstrating enhanced relevance filtering, while recall slightly decreased, indicating a focus on accuracy.
Significance
This work advances scholarly search by integrating citation-aware exploration with multi-agent coordination, addressing limitations of static keyword-based systems. It models researcher-like behaviors, enabling deeper, more accurate literature discovery across disciplines. The framework's scalability and interpretability facilitate widespread adoption in academic platforms, significantly reducing manual filtering and enhancing research efficiency. The creation of SPARBench provides a standardized, realistic evaluation environment, fostering further innovation in intelligent academic retrieval.
Technical Contribution
The paper introduces a modular, training-free multi-agent system that combines RefChain citation expansion, query understanding, and dynamic query evolution. It innovates by embedding expert-annotated benchmarks, enabling comprehensive evaluation. The architecture allows flexible integration of multiple data sources and iterative refinement, setting a new standard for deep scholarly retrieval systems. The approach balances precision and recall effectively, demonstrating robustness across datasets.
Novelty
This is the first work to embed RefChain within a multi-agent framework for recursive citation expansion in academic search. It combines multi-source, multi-turn query evolution with expert-validated benchmarks, offering a novel, scalable solution for complex scholarly queries. Unlike prior methods relying on supervised RL or static models, SPAR's modular design and real-world evaluation mark a significant step forward in the field.
Limitations
- The system's understanding of highly ambiguous or multi-faceted queries remains imperfect, potentially missing relevant literature.
- Multi-agent coordination introduces complexity, affecting real-time performance and system stability.
- Dependence on LLMs' current capabilities, which may carry biases or outdated knowledge, limits accuracy in rapidly evolving fields.
Future Work
Future efforts will focus on multi-turn dialogue integration, deeper citation chain exploration, and personalized user feedback loops. Expanding across more disciplines and larger datasets will enhance generalization. Incorporating knowledge graphs and multimodal data could further improve semantic understanding and retrieval depth.
AI Executive Summary
Academic literature retrieval has historically relied on keyword matching, which struggles with complex, multi-intent queries common in modern research. Traditional systems like Google Scholar excel at simple searches but falter when faced with nuanced questions requiring contextual understanding and citation-based exploration. Recent advances in large language models (LLMs) have introduced capabilities such as query rewriting and document ranking, yet these are often limited to single-turn interactions, lacking the iterative, citation-aware exploration that researchers perform naturally.
To bridge this gap, the authors propose SPAR, a novel multi-agent framework inspired by how researchers follow citation trails to deepen their understanding. SPAR integrates five specialized agents: Query Understanding, Retrieval, Query Evolver, Judgment, and Reranker. The core innovation lies in combining RefChain, which simulates citation traversal, with dynamic query evolution, enabling the system to iteratively refine and expand search results. This design mimics scholarly behavior, allowing the system to explore beyond initial keyword matches, capturing relevant literature across multiple sources such as arXiv, OpenAlex, Semantic Scholar, and PubMed.
The framework leverages state-of-the-art LLMs like Qwen3-32B for intent detection, keyword extraction, and relevance scoring, all without requiring training. The process begins with understanding the user’s complex query, identifying intent, domain, and temporal constraints. It then performs multi-source retrieval, expanding the candidate set via citation links while controlling depth to maintain relevance. Query evolution generates new search angles, further enriching the results. The Judgment agent filters relevant papers, and the Reranker sorts them based on authority and recency, producing a high-quality, tailored list.
Empirical evaluations on AutoScholar and SPARBench demonstrate that SPAR outperforms existing methods significantly, with F1 improvements of 56% and 23%, respectively. These results validate the system’s ability to balance precision and recall, handle complex queries, and adapt across disciplines. The creation of SPARBench offers a standardized, expert-annotated benchmark for realistic academic search evaluation, fostering future research.
This work marks a substantial step towards intelligent, citation-aware scholarly search systems that can support researchers in navigating the ever-expanding scientific literature. Future directions include multi-turn dialogue, deeper citation exploration, and broader domain coverage, aiming to make academic discovery more efficient, accurate, and accessible.
Deep Dive
Abstract
Recent advances in large language models (LLMs) have opened new opportunities for academic literature retrieval. However, existing systems often rely on rigid pipelines and exhibit limited reasoning capabilities. We introduce SPAR, a multi-agent framework that incorporates RefChain-based query decomposition and query evolution to enable more flexible and effective search. To facilitate systematic evaluation, we also construct SPARBench, a challenging benchmark with expert-annotated relevance labels. Experimental results demonstrate that SPAR substantially outperforms strong baselines, achieving up to +56% F1 on AutoScholar and +23% F1 on SPARBench over the best-performing baseline. Together, SPAR and SPARBench provide a scalable, interpretable, and high-performing foundation for advancing research in scholarly retrieval. Code and data will be available at: https://github.com/xiaofengShi/SPAR