DeepXiv-SDK: An Agentic Data Interface for Scientific Literature

The exponential increase in scientific publications over the past decade has led to a pressing need for more efficient literature management tools. Early systems like Google Scholar and Microsoft Academic provided keyword-based search and metadata retrieval, but struggled with long documents and complex structures. Recent efforts such as ar5iv and AlphaXiv attempted to improve user experience by converting papers into HTML with enriched views, yet lacked standardized, reusable interfaces for AI agents. The advent of large language models (LLMs) has intensified the demand for structured, accessible scientific data, prompting research into document normalization, signal extraction, and multi-level retrieval. Despite these advances, existing tools remain fragmented, often relying on ad hoc parsing, which hampers automation, reproducibility, and scalability. The challenge remains to develop a unified, scalable infrastructure that can transform heterogeneous scientific documents into structured, tool-callable entities, supporting complex AI workflows.

Current scientific literature access methods are inefficient for AI-driven workflows. Raw PDFs and HTML pages require extensive parsing, which is brittle and inconsistent across formats. This leads to high computational costs, token wastage, and unreliable evidence retrieval. Moreover, existing systems lack standardized, multi-level interfaces that allow incremental, cost-aware access, forcing models to reason over noisy, unstructured text. These limitations hinder the deployment of autonomous scientific agents capable of multi-step reasoning, evidence verification, and knowledge synthesis. The core problem is to create a robust, scalable, structured, and progressive data interface that can serve as a reliable foundation for AI applications in scientific research, enabling low-cost screening, targeted reading, and evidence-based decision-making.

DeepXiv-SDK introduces a novel three-layer architecture that fundamentally redefines scientific literature access. The data layer employs deterministic pipelines to convert heterogeneous PDFs and HTMLs into a canonical JSON schema, capturing metadata, structural signals, and auxiliary indicators like citation counts. The service layer offers a RESTful API supporting multi-attribute, hybrid retrieval, and progressive views—header, section, and evidence—allowing low-cost screening and targeted access. The application layer encapsulates SDKs and intelligent agents that orchestrate multi-step workflows, including deep search and evidence-backed QA, with high concurrency. This integrated approach surpasses prior tools by enabling structured, reusable, and cost-aware access, tailored for AI automation. It bridges the gap between unstructured scientific documents and AI models, facilitating scalable, reliable, and efficient workflows.

• �� Data Layer:
• Input: arXiv ID
• Metadata retrieval via OAI-PMH
• Source acquisition: HTML preferred, fallback to PDF
• PDF normalization: MinerU converts PDF to Markdown
• HTML content extraction: main content parsing
• Structure detection: heading cues, formatting patterns
• JSON schema generation: paper metadata, section hierarchy
• Signal extraction: token counts, semantic summaries, resource links, citation/social signals
• Signal enrichment: TL;DRs, keywords, resource validation
• Storage: multiple views (overview, section, evidence) with provenance

• �� Service Layer:
• REST API endpoints supporting:
• �� Header-first view for screening
• �� Section view for targeted reading
• �� Evidence view for verification
• �� Attribute-conditioned hybrid retrieval
• Authentication, caching, on-demand loading
• SDK and CLI clients for integration

• �� Application Layer:
• SDK wrapping REST calls with deterministic functions
• Built-in agent for multi-step workflows
• Workflow examples: deep search, evidence extraction
• High concurrency support for large-scale deployment

This architecture ensures modularity, reusability, and scalability, enabling AI agents to perform complex, cost-aware literature analysis efficiently.

The evaluation involved two primary tasks: agentic paper search and deep research QA. The dataset comprised 50 multi-constraint queries targeting unique papers and 47 complex QA queries requiring evidence synthesis, all based on the latest arXiv corpus. Baselines included Google Scholar, AlphaXiv, and other academic search tools. Metrics measured were recall@1, latency, token consumption, and answer correctness. In retrieval tasks, DeepXiv achieved an average recall@1 of 78.26%, with an average latency of 43.9 ms, outperforming baselines by a significant margin. The system's hybrid retrieval and progressive views reduced token usage by over 60%. In QA tasks, answer accuracy improved by approximately 15%, with token savings exceeding 60%. The system maintained stable performance under high concurrency, processing millions of requests daily, demonstrating robustness and scalability.

DeepXiv demonstrated superior performance in both retrieval accuracy and efficiency. Its ability to leverage structured, multi-level views allowed for rapid filtering and targeted reading, significantly reducing computational costs. The hybrid retrieval mechanism effectively balanced lexical and semantic signals, resulting in higher precision. The latency benchmarks confirmed that the system could handle large-scale, real-time applications with response times within hundreds of milliseconds. These results validate DeepXiv’s potential to transform scientific literature management, enabling AI agents to operate more effectively and efficiently in complex research workflows.

DeepXiv-SDK is applicable in automated literature review, evidence-based scientific inquiry, and AI-powered research assistants. It allows researchers to quickly identify relevant papers, verify claims, and synthesize knowledge with minimal manual effort. Academic institutions can integrate it into their digital libraries for enhanced search and retrieval. Industry sectors such as pharmaceuticals, materials science, and AI research benefit from rapid access to verified evidence, accelerating innovation cycles. Its scalable architecture supports integration into large-scale AI systems, enabling autonomous scientific discovery and decision-making.

The current implementation primarily relies on arXiv data, limiting coverage to preprints and open-access repositories. Extending to proprietary journals and multi-lingual documents requires additional adaptation. Structure recovery algorithms may face challenges with poorly formatted or scanned PDFs, affecting accuracy. High concurrency deployment demands ongoing optimization of computational resources and cost management. Moreover, semantic understanding beyond structural signals, such as deep reasoning or inference, remains an open area for future development. Addressing these limitations will be crucial for broader adoption and robustness.