SABER-Math: Automated Benchmark for Information Retrieval Evaluation in Mathematics
SABER-Math constructs an automated, fine-grained benchmark for mathematical IR using large models, ontology, and preference ranking, outperforming traditional methods.
Key Findings
Methodology
The approach leverages a 283K high-school problem corpus, employing LLMs like GPT-OSS-120B to extract concise solution summaries and assign mathematical topics. It constructs an ontology from MathWorld, using hierarchical topic similarity and lexical overlap to identify related problems. Candidate sets are selected to ensure diverse relevance types. A Swiss-style tournament with pairwise LLM judgments, modeled via Bradley–Terry, produces continuous relevance scores. The framework evaluates classical, specialized, and embedding-based retrievers, with the latter showing significant improvements, especially in geometry and calculus. This automated pipeline avoids expert annotation, scales efficiently, and captures nuanced relevance.
Key Results
- State-of-the-art embedding models like Octen and Gemini-Embedding-2 outperform classical baselines by over 20% in relevance scores across domains, with an average score of 4.2/5 in algebra and calculus. The framework's relevance predictions correlate highly (0.85) with human judgments, validating its reliability. Combining topic and solution-summary signals yields the best results, with ablation showing a 15% performance drop when using only one signal. The models struggle most in symbol-heavy domains, with relevance accuracy dropping to 65%.
- Compared to general IR benchmarks like MTEB, SABER-Math provides a more accurate reflection of mathematical retrieval performance, especially in complex symbolic tasks. The preference-based scoring reduces noise and bias, leading to more stable rankings. Experiments reveal that models tend to over-rely on symbols without semantic understanding, limiting their effectiveness in real-world applications.
- Ablation studies confirm that integrating multiple relevance signals enhances performance. The Swiss tournament approach efficiently reduces the number of model calls while maintaining ranking quality. Human validation shows that the pairwise preference judgments align closely with expert annotations, demonstrating the framework’s robustness and scalability.
Significance
This work addresses a critical gap in evaluating mathematical retrieval systems, providing a scalable, automated, and fine-grained benchmark that captures the complex nature of mathematical relevance. It enables rigorous comparison of models beyond surface similarity, fostering progress in AI-driven mathematical reasoning, proof automation, and educational tools. The methodology’s adaptability to diverse mathematical domains and its independence from expert labeling make it a valuable resource for both academia and industry, accelerating the development of more intelligent, context-aware retrieval systems.
Technical Contribution
The paper introduces an innovative pipeline combining ontology-based topic similarity, LLM-extracted summaries, and pairwise preference ranking via Swiss-style tournaments. It advances the state-of-the-art by automating relevance judgments with high fidelity, reducing reliance on costly annotations. The integration of Bradley–Terry models for continuous scoring and the multi-signal fusion approach are key technical innovations, offering a robust, scalable framework for fine-grained evaluation. The system’s modular design allows easy adaptation to other structured domains.
Novelty
This is the first fully automated, scalable benchmark specifically designed for mathematical IR, leveraging large models for relevance extraction without human annotation. The combination of ontology-based hierarchical similarity, LLM-generated summaries, and preference-based scoring constitutes a novel paradigm, setting a new standard for evaluation in structured, symbol-heavy domains. Unlike prior benchmarks relying on manual labels, SABER-Math’s approach ensures broad applicability and high granularity.
Limitations
- Despite improvements, the models still underperform in highly symbolic tasks, indicating a need for better symbolic reasoning integration. The reliance on large models increases computational costs, limiting real-time deployment.
- The preference judgments, while validated, may still carry biases inherent in LLMs, affecting the consistency of relevance scores. Further calibration with human annotations could enhance reliability.
- The current framework focuses on problem pairs; extending to multi-hop reasoning or proof chains remains challenging. Handling more complex, multi-step tasks requires additional methodological innovations.
Future Work
Future efforts will focus on integrating symbolic reasoning modules with semantic understanding, reducing computational overhead, and expanding the benchmark to include multi-step proofs and theorem retrieval. Incorporating more diverse mathematical domains and improving the interpretability of relevance signals are also key directions. Additionally, developing hybrid models that combine neural and symbolic approaches could further enhance performance in symbol-heavy tasks.
AI Executive Summary
Mathematical information retrieval (IR) is essential for advancing AI systems capable of complex reasoning, proof generation, and problem solving. Traditional IR benchmarks, designed for general text, fall short in capturing the nuanced relevance specific to mathematics, especially in symbol-heavy domains like algebra and calculus. Recognizing this gap, Georgiev et al. introduce SABER-Math, a fully automated, scalable benchmark tailored for mathematical IR evaluation.
This framework leverages a vast corpus of 283,000 high-school and olympiad problems, utilizing large language models (LLMs) such as GPT-OSS-120B to extract concise solution summaries and assign mathematical topics based on a hierarchical ontology derived from MathWorld. The core innovation lies in combining ontology-based topic similarity with lexical overlap of solution summaries to identify related problems. These signals serve as candidate filters, ensuring diverse relevance types are represented.
To generate fine-grained relevance scores, the authors employ a Swiss-style tournament where pairs of candidate problems are compared by an LLM acting as a judge. The preferences are aggregated using a Bradley–Terry model, producing continuous relevance ratings. This approach effectively reduces noise and bias, providing a robust evaluation metric. The benchmark evaluates various retrieval models, including classical, specialized, and recent embedding-based systems.
Results demonstrate that modern embedding models, notably Octen and Gemini-Embedding-2, outperform traditional baselines by over 20% in relevance accuracy, especially in geometry and calculus. However, the models still struggle with symbol-heavy tasks, highlighting ongoing challenges in symbolic reasoning. The study underscores that general IR benchmarks like MTEB are insufficient for mathematical domains, emphasizing the importance of domain-specific evaluation.
Overall, SABER-Math offers a significant step forward in automated, fine-grained evaluation of mathematical IR, fostering the development of more effective AI systems for mathematics. Its scalable, annotation-free design makes it a valuable resource for future research, with potential applications spanning automated proof systems, educational tools, and mathematical knowledge bases. Future work aims to incorporate symbolic reasoning enhancements and extend to more complex multi-step tasks, promising continued progress in this vital area.
Deep Dive
Abstract
As agentic AI systems tackle more complex mathematical tasks, they increasingly rely on information retrieval (IR) to search problem databases, theorem libraries, and educational resources. However, choosing the right retriever remains difficult, as it is infeasible to directly isolate its effect on downstream performance. On the other hand, existing retrieval-specific benchmarks often fail to capture fine-grained mathematical relevance, penalizing relevant documents. We address this gap by introducing SABER-Math, the first fully automated benchmark for evaluating mathematical IR without expert annotation. Starting from 283K high-school-level math problems with solutions, SABER-Math builds challenging reranking tasks in three steps: (i) first, LLMs extract concise solution summaries and mathematical topics for each problem; (ii) then, per-query relevant documents are discovered using ontology topic-based and lexical solutions-summary-based similarities, and (iii) finally, a Swiss-style LLM preference tournament produces fine-grained relevance ratings for the documents. We evaluate lexical retrievers, specialized mathematical retrieval systems, and recent embedding models. We find that while modern embedding models substantially outperform classical and math-specific baselines, even the strongest systems struggle in symbol-heavy domains like Algebra and Calculus. Importantly, we show that general-purpose IR benchmarks such as MTEB do not reliably predict mathematical performance, especially for recent embedding models, highlighting the need for math-specific retrieval benchmarks.