PaperArena: An Evaluation Benchmark for Tool-Augmented Agentic Reasoning on Scientific Literature

The rapid expansion of scientific literature, especially in AI, has driven the development of automated understanding and reasoning systems. Early benchmarks like PubMedQA and CharXiv focused on localized tasks such as fact extraction and concept explanation. With the advent of large language models (LLMs), research shifted toward knowledge-intensive tasks, including structured data interpretation, scientific reasoning, and literature synthesis. Existing datasets primarily evaluate single-task performance, often ignoring the multi-modal and multi-document nature of real research workflows. Recent efforts incorporate multimodal understanding, peer review simulation, and domain-specific challenges, but lack comprehensive evaluation of multi-tool, cross-paper reasoning capabilities. This gap limits progress in building autonomous scientific agents capable of complex reasoning across diverse content types and sources.

Despite advances, current LLMs struggle with complex scientific reasoning involving multiple steps, diverse content modalities, and cross-referencing multiple papers. Existing benchmarks do not adequately reflect real research scenarios, which require integrating textual, graphical, and tabular data, while coordinating multiple external tools. The bottleneck lies in models' inability to efficiently manage multi-modal inputs, formulate coherent reasoning plans, and invoke appropriate tools without redundancy. This results in low accuracy, high tool invocation cost, and poor interpretability. Addressing these issues is critical for developing autonomous AI systems that can assist in scientific discovery, literature review, and hypothesis generation, especially as the volume of scientific data continues to grow exponentially.

The key innovations of PaperArena include:

• �� Cross-paper reasoning: questions require synthesizing information from multiple papers, reflecting real research workflows.
• �� Multi-modal understanding: tasks involve interpreting charts, tables, and textual data simultaneously.
• �� Multi-tool orchestration: models must invoke diverse tools like PDF parsers, graph analyzers, and database queriers.
• �� Expert-curated high-difficulty questions: ensuring challenge and relevance.
• �� Latent ability diagnostics via IRT: providing insights into model strengths and weaknesses.

These innovations collectively enable a more realistic and comprehensive evaluation of AI's scientific reasoning capabilities, surpassing prior benchmarks limited to single-task or single-document scenarios.

• �� Curate a large corpus of 14,435 open-access AI papers, applying hierarchical sampling (K-Medoids + FPS) to select 100 representative papers.
• �� Convert PDFs into structured markdown using MinerU, preserving sections, citations, and figures.
• �� Generate 2,875 candidate questions via three advanced LLMs, structured to cover multi-step, multimodal, cross-paper, and database tasks.
• �� Expert annotators filter, rewrite, and label questions with answers, tool chains, difficulty levels, and question types.
• �� Develop a modular platform supporting multimodal parsing, context retrieval, and program execution.
• �� Design multi-task, multi-tool reasoning workflows where models formulate plans, invoke tools, and synthesize results.
• �� Evaluate nine top LLMs in single- and multi-agent setups, measuring accuracy, reasoning steps, tool efficiency, and latent ability via IRT.
• �� Analyze reasoning traces, tool usage patterns, and model performance across difficulty levels to identify bottlenecks and strengths.

The evaluation involves nine leading LLMs, including Gemini 2.5 Pro, GPT-4.1, and Qwen3-235B, tested on 784 high-difficulty questions across varying task types and difficulty levels. Both single- and multi-agent workflows are examined, with metrics including accuracy, reasoning steps, tool invocation efficiency, and latent ability via IRT. Experiments include ablation studies on tool usage imbalance, reasoning chain length, and multi-modal understanding. Human expert performance (83.5%) serves as a benchmark. Results show significant performance gaps, especially on complex tasks, highlighting the need for better planning and tool management strategies. The platform enables detailed diagnostics of model behavior and failure modes.

The best model, Gemini 2.5 Pro, achieves 38.78% average accuracy, with only 18.47% on hard questions, far below human experts at 83.5%. Models tend to overuse general-purpose tools, leading to inefficiency. Multi-agent systems outperform single-agent setups, especially on complex tasks, validating the centralized management approach. IRT analysis reveals that models' latent abilities are mismatched with task difficulty, exposing core limitations in multi-step reasoning and cross-paper integration. Tool usage analysis shows a bias towards web search and code execution, often invoking unnecessary tools, which hampers efficiency. These findings underscore the importance of balanced tool invocation and advanced planning strategies for future improvements.

This benchmark can be used to develop AI assistants for literature review, automated hypothesis generation, and scientific data analysis. It supports researchers in quickly synthesizing information from multiple sources, interpreting complex graphical data, and interfacing with external databases. Industry applications include automated research report generation, patent analysis, and academic recommendation systems. As models improve in multi-modal and multi-tool reasoning, they will increasingly support autonomous scientific discovery, reducing manual effort and accelerating innovation across disciplines.

Models still exhibit inefficiencies, often invoking unnecessary tools, which increases computational costs. The platform relies on existing tools and models, limiting scope for innovation. The static evaluation environment does not fully capture dynamic research workflows, such as real-time data updates or iterative hypothesis testing. Future work should focus on optimizing tool invocation, enhancing multi-modal understanding, and reducing resource consumption. Additionally, integrating real-time data streams and improving multi-agent coordination will be necessary to approach autonomous scientific reasoning in real-world settings.