MathArena: Evaluating LLMs on Uncontaminated Math Competitions
MathArena leverages newly released math competitions for real-time, uncontaminated evaluation of LLM reasoning and proof-writing, surpassing static benchmarks.
Key Findings
Methodology
This work introduces MathArena, a dynamic benchmark utilizing real-time, unreleased competition problems to evaluate LLMs’ reasoning and proof capabilities. By testing models immediately after problem release, it prevents data contamination. The framework combines automated parsing, manual proof evaluation, and multiple response averaging, ensuring fair and reliable assessment. Covering seven competitions with 162 problems across algebra, geometry, and number theory, model performance is measured via accuracy, proof quality, and statistical confidence intervals. Results show that top models outperform human top 1% in some competitions but reveal contamination issues in AIME 2024, while demonstrating strong reasoning in CMIMC 2025 and proof capabilities in IMO 2025.
Key Results
- Models like GPT-5, GROK 4, and GEMINI-2.5-PRO achieved over 90% accuracy on CMIMC 2025, indicating robust reasoning. In AIME 2024, some models scored above human levels due to contamination, with QWQ-PREVIEW-32B exceeding expected performance by nearly 60%. In IMO 2025, proof-writing accuracy was below 40%, highlighting significant room for improvement. Overall, evaluation across 50+ models confirms MathArena’s effectiveness in uncontaminated, real-time assessment.
- The platform’s dynamic approach effectively detects contamination, as evidenced by performance discrepancies between 2024 and 2025 competitions, and provides a reliable measure of true reasoning ability.
- The results demonstrate that current models have made substantial progress in solving high-level math problems, but proof-writing remains a challenging frontier, requiring further research.
Significance
This study addresses core limitations of static, contaminated benchmarks by establishing a real-time, uncontaminated evaluation framework. It ensures fair comparison of models’ reasoning and proof skills, fostering genuine progress in mathematical AI. The approach enhances reproducibility, transparency, and relevance, setting a new standard for future benchmarks. It also bridges the gap between academic research and practical deployment, enabling the development of AI systems capable of rigorous mathematical reasoning and formal proof generation, which are essential for advancing AI’s role in scientific discovery and education.
Technical Contribution
The paper presents a comprehensive pipeline integrating problem extraction, multi-model response generation, automated parsing, manual proof evaluation, and statistical analysis. It introduces a novel dynamic benchmarking paradigm that updates with new competitions, significantly reducing contamination risks. The framework supports both answer-based and proof-based problems, with automated answer parsing using sympy and rule-based methods, and proof evaluation via expert graders. The statistical methods include confidence interval estimation for model ranking and accuracy, ensuring robust performance assessment. This work establishes a scalable, transparent, and reproducible evaluation environment for mathematical reasoning.
Novelty
This is the first work to implement a real-time, dynamic benchmark based on unreleased, high-quality math competitions, effectively eliminating data contamination. Unlike static datasets, it continuously adapts to new problems, providing a fair and current assessment of model capabilities. Its hybrid automated-human evaluation approach, especially for proof-writing, offers a significant advancement over prior static or private benchmarks, which often suffer from reproducibility and transparency issues.
Limitations
- Despite progress, proof-writing accuracy remains below 40%, indicating that models still struggle with formal reasoning and logical rigor.
- Automated proof verification is limited; reliance on human graders introduces subjectivity and scalability issues.
- Potential online leakages of new competition problems could introduce contamination risks, requiring ongoing vigilance.
Future Work
Future efforts will focus on expanding the competition set, improving automated proof verification, and integrating multi-modal tasks such as geometric diagrams and code-based problems. Developing more sophisticated parsing and reasoning algorithms, along with larger, more diverse datasets, will further enhance evaluation robustness. Additionally, fostering community contributions and open-sourcing tools will accelerate progress, aiming for models capable of complex reasoning and formal proof generation at near-human levels.
AI Executive Summary
MathArena pioneers a novel evaluation paradigm for large language models by harnessing the real-time release of high-quality math competition problems. Unlike traditional benchmarks, which often suffer from data contamination and static limitations, MathArena continuously updates with unreleased problems, ensuring uncontaminated, fair assessment of models’ reasoning and proof capabilities. This approach addresses longstanding issues in the field, such as the difficulty of measuring genuine understanding versus memorization, and the challenge of evaluating proof-writing skills.
The framework involves immediate testing of models after problem release, automated parsing of answers, manual verification of proofs, and statistical analysis to derive robust performance metrics. Covering seven competitions with 162 problems, the platform provides a comprehensive, transparent, and reproducible environment for benchmarking. Results show that top models like GPT-5 and GROK 4 surpass human performance in some competitions, yet reveal ongoing challenges in proof generation, with scores below 40% in IMO 2025.
This work significantly advances the state-of-the-art in mathematical AI evaluation, offering a scalable, fair, and up-to-date standard. It not only fosters genuine progress in reasoning and proof capabilities but also sets a foundation for future research, including expanding problem diversity, improving automated proof verification, and integrating multi-modal reasoning tasks. Ultimately, MathArena aims to accelerate the development of AI systems that can truly understand and generate rigorous mathematical knowledge, impacting both academia and industry.
Deep Dive
Abstract
The rapid advancement of reasoning capabilities in large language models (LLMs) has led to notable improvements on mathematical benchmarks. However, many of the most commonly used evaluation datasets (e.g., AIME 2024) are widely available online, making it difficult to disentangle genuine reasoning from potential memorization. Furthermore, these benchmarks do not evaluate proof-writing capabilities, which are crucial for many mathematical tasks. To address this, we introduce MathArena, a new benchmark based on the following key insight: recurring math competitions provide a stream of high-quality, challenging problems that can be used for real-time evaluation of LLMs. By evaluating models as soon as new problems are released, we effectively eliminate the risk of contamination. Using this framework, we find strong signs of contamination in AIME 2024. Nonetheless, evaluations on harder competitions, such as CMIMC 2025, demonstrate impressive reasoning capabilities in top-performing models. MathArena is also the first benchmark for proof-writing capabilities. On IMO 2025, top models achieve slightly less than 40%, demonstrating both notable progress and significant room for improvement. So far, we have evaluated over $50$ models across seven competitions, totaling $162$ problems. As an evolving benchmark, MathArena will continue to track the progress of LLMs on newly released competitions, ensuring rigorous and up-to-date evaluation of mathematical reasoning.