NL2AGBench: Benchmarking LLM Auto-Formalization for AlphaGeometry
NL2AGBench evaluates large models’ ability to convert English geometry problems into AlphaGeometry formal language, achieving over 80% success in closed models.
Key Findings
Methodology
This study introduces NL2AGBench, a benchmark combining AlphaGeometry’s execution verification with LLM-based translation. It assesses 10 models, including GPT-4, Gemini-3.1, and Llama, across parameter scales. The framework prompts models with official syntax definitions, rules, and examples, then evaluates generated formalizations by executing them in AlphaGeometry. Errors are categorized into syntax, logic, and value errors, providing detailed diagnostics. Strategies like few-shot prompting, fine-tuning, and human hints are employed to improve translation accuracy. The success criterion is that the generated formalization executes without errors and preserves geometric semantics, ensuring practical utility.
Key Results
- Closed-source models such as GPT-4 and Gemini-3.1 achieve over 80% executable translation success, whereas open-source models like Llama3.1 reach only 23%. Few-shot prompting significantly boosts open-source performance, with Qwen3:235B rising from 12.5% to 45.8%. Error analysis shows syntax errors dominate, with logical errors also present. Fine-tuning and human guidance further improve success rates, especially in preserving geometric constraints. These results demonstrate the gap between proprietary and open models and highlight the importance of prompting strategies.
- The error taxonomy reveals that syntax errors, such as incorrect clause arguments or misplaced tokens, account for most failures. Logic errors, like misinterpreted geometric relations, also contribute. Strategies like few-shot prompting reduce syntax errors by providing model examples, while human hints help clarify ambiguous cases. The experiments confirm that targeted prompts and guidance are effective, especially for large open-source models, but still fall short of the performance of top proprietary models. The evaluation framework ensures that success is measured by functional execution, not just textual similarity.
- Overall, the experiments validate the effectiveness of execution-based verification and prompt engineering. They show that current models can achieve high success rates with proper guidance but still face challenges in complex geometric reasoning. The work emphasizes the need for better geometric understanding within models and suggests future directions including multimodal approaches and enhanced training data to close the performance gap.
Significance
This research establishes a standardized benchmark for evaluating the auto-formalization of natural language geometry problems, a crucial step toward automating geometric theorem proving. By integrating execution verification, it ensures that generated formalizations are not only syntactically correct but also geometrically meaningful. The findings highlight the strengths of proprietary models and the potential of open-source models with prompt engineering, guiding future development. The work addresses a long-standing bottleneck in neuro-symbolic AI, enabling scalable, reliable translation from natural language to formal geometry representations. This has broad implications for AI-assisted education, automated theorem proving, and scientific reasoning, advancing the frontier of AI in mathematical domains.
Technical Contribution
The paper presents NL2AGBench, a novel benchmark integrating execution-based evaluation for natural language to AlphaGeometry formalization. It systematically categorizes translation failures into syntax, logic, and value errors, providing detailed diagnostics. The framework employs multi-strategy optimization—few-shot prompts, supervised fine-tuning, and human-guided hints—to enhance model performance. The experimental results demonstrate that prompt engineering significantly improves open-source model success rates, narrowing the gap with proprietary models. The approach emphasizes practical validation through execution, ensuring the formalizations are usable for theorem proving. This work advances the state-of-the-art in neuro-symbolic AI, providing a comprehensive evaluation methodology and actionable insights for future research.
Novelty
This is the first comprehensive benchmark specifically targeting the translation of natural language geometry problems into AlphaGeometry-compatible syntax, with an emphasis on execution-based validation. It introduces a detailed error taxonomy for formalization failures, enabling precise diagnosis and targeted improvements. The integration of multi-strategy enhancement—prompting, fine-tuning, and human hints—demonstrates practical pathways to boost open-source model performance. Unlike prior work focusing solely on reasoning accuracy, this study emphasizes the correctness of symbolic formalization, bridging a critical gap in neuro-symbolic AI research. Its systematic evaluation and diagnostic framework set a new standard for future auto-formalization benchmarks.
Limitations
- Models still struggle with complex spatial relationships and implicit geometric constraints, leading to semantic inaccuracies despite syntactic correctness.
- The reliance on large prompts and fine-tuning increases computational costs, limiting scalability for real-time applications.
- Current models exhibit instability in maintaining geometric consistency and symbolic correctness, especially under complex configurations. Further research is needed to improve geometric understanding and reasoning robustness.
Future Work
Future research will explore multimodal models integrating visual geometric diagrams with language inputs, aiming to enhance spatial reasoning. Developing more efficient prompting and fine-tuning techniques will reduce computational overhead. Extending the benchmark to broader mathematical domains and improving models’ intrinsic geometric understanding are key directions. Additionally, integrating symbolic reasoning modules with neural models could further improve accuracy and robustness, pushing the boundaries of automated geometric theorem proving.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房做菜，食谱告诉你需要哪些食材、怎么放、用什么工具，但你还得用正确的刀、锅、调料，才能做出美味的菜。自然语言的几何题就像这个食谱，描述了问题的内容，但要让机器人帮你解题，就得把这些描述变成它能懂的“厨具”——符号语言。这个转化过程很复杂，因为题目里有很多空间关系和隐含信息，不能只看文字。研究人员设计了一个测试，让模型把英语描述转成符号语言，然后用专门的几何软件验证它是否正确。结果显示，大公司做的模型（比如GPT-4）能成功转化超过80%的题目，而开源模型（像Llama）成功率只有20多一点。通过多次练习和提示，模型的表现逐步改善。这个工作就像教机器人用正确的厨具做菜，让它变得更聪明，将来可以帮学生学习几何、自动解题，甚至帮科学家验证几何猜想。
ELI14 Explained like you're 14
想象你在学校参加科学比赛，题目用英语写得很复杂，描述了一个几何问题。你需要把这个英语题转成一种特殊的符号语言，才能让电脑帮你算。可是，这很难，因为英语里有很多隐含的空间关系和细节，不能随便写。科学家们想让电脑自己学会把英语题变成符号语言，就像教一个新手厨师用正确的刀和锅一样。于是，他们设计了一个测试，让电脑尝试转化，然后用专门的几何软件验证转化是否正确。研究发现，大公司做的电脑模型（比如GPT-4）能成功转化超过80%的题目，而开源模型（像Llama）成功率只有20多一点。通过给模型更多例子和提示，效果会变得更好。这个研究很重要，因为它让电脑更聪明，能自己理解复杂的几何题，将来可以帮助学生学习、自动解题，甚至帮科学家验证几何猜想。
Abstract
Recent advances in large language models (LLMs) have demonstrated strong capabilities in natural language understanding and mathematical reasoning. However, their ability to translate informal mathematical problems into formal representations remains underexplored. This limitation is particularly important for neuro-symbolic geometry systems such as AlphaGeometry, whose theorem-proving engine requires inputs in a specialized domain-specific language (DSL). Although AlphaGeometry achieves near-IMO gold-medalist performance, manually converting natural-language problems into its formal syntax remains a significant usability bottleneck. To address this challenge, we introduce the Natural Language to AlphaGeometry Benchmark (NL2AGBench), which evaluates LLMs in translating English geometry problems into AlphaGeometry-compatible formal representations. NL2AGBench uses execution-based verification within AlphaGeometry to assess translation quality rather than relying solely on textual similarity. We evaluate ten state-of-the-art open- and closed-source LLMs across multiple parameter scales and analyze executable translation accuracy, syntactic correctness, and error characteristics. Our experiments reveal a substantial performance gap between closed- and open-source models: leading closed-source models achieve executable translation rates above 80%, while even the largest open-source models struggle to consistently preserve geometric constraints and produce valid formalizations. We introduce an error taxonomy distinguishing syntax and logic errors and investigate mitigation strategies, including few-shot prompting, fine-tuning, and human-guided hinting, which yield measurable improvements across multiple model families.