Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement Formalization
Proposes a faithfulness evaluation framework combining Lean compilation and semantic judging, revealing a 30% gap between compilation success and semantic accuracy on 400 statements.
Key Findings
Methodology
The approach integrates Lean compiler verification, cross-model semantic judgment using GPT-5.2 and Gemini-2.5-Pro, and expert calibration. A curated dataset of 400 graduate-level mathematical statements across analysis, topology, and algebra was used. The evaluation employs a multi-layered protocol: first filtering by compilation success, then semantic scoring by two independent LLM judges, with consensus thresholds set at scores ≥9. The study employs a 23-factor factorial design to analyze the impact of tools—expert drafting, Mathlib search, and elaboration feedback—on validity and faithfulness. This comprehensive pipeline enables detailed attribution of each intervention’s contribution to the overall performance.
Key Results
- The full system achieves 89.5% compilation but only 60.5% faithfulness, exposing a 29% gap where statements compile but lack semantic fidelity. Human audits confirm that 96% of positive consensus outputs are faithful, while 82.4% of negative consensus outputs are semantic failures. Existing models without tool augmentation perform significantly worse, with faithfulness below 30%. The analysis highlights that compilation success alone is insufficient for trustworthy formalization, emphasizing the need for semantic verification.
- Tool interventions show that elaboration feedback is the most impactful for validity, increasing faithful outputs by approximately 30%, but also enlarging the semantic gap. Search tools mainly improve grounding and selectivity, while expert drafting becomes largely redundant once feedback and grounding are available. The factorial analysis demonstrates that different tools influence validation boundaries differently, with feedback expanding validity but risking semantic drift.
- Comparative analysis across models indicates that the integrated tool-augmented system outperforms pure language models by a large margin, achieving over 60% faithfulness, whereas standalone models hover below 30%. This underscores the importance of tool integration and multi-modal verification in automatic formalization pipelines.
Significance
This work pioneers a comprehensive evaluation framework for natural-language-to-Lean statement formalization, emphasizing semantic faithfulness beyond mere compilation success. By revealing a substantial gap between syntactic correctness and semantic fidelity, it addresses a core challenge in trustworthy AI-driven mathematics. The methodology combines compiler validation, large language model judgment, and expert calibration, setting a new standard for assessing formalization systems. The insights into tool contributions guide future system design, emphasizing the importance of elaboration feedback and grounding strategies. The findings have broad implications for AI-assisted mathematical research, education, and automated theorem proving, fostering more reliable and interpretable AI systems in formal mathematics.
Technical Contribution
The paper introduces a novel multi-model, multi-tool evaluation framework that combines Lean compilation verification with cross-model semantic judgment, grounded in a curated dataset and expert calibration. The core technical innovation is the factorial design analysis of tool effects—expert drafting, Mathlib search, and elaboration feedback—on validation boundaries, enabling precise attribution of improvements. The methodology advances the state-of-the-art by providing a scalable, high-precision metric for semantic faithfulness, moving beyond traditional compile success metrics. It also demonstrates how tool synergy can significantly enhance both validity and faithfulness, offering a blueprint for future AI-driven formalization pipelines.
Novelty
This is the first systematic study integrating Lean compiler verification with large language model semantic judgment to evaluate statement faithfulness at scale. Unlike prior work focusing solely on proof success or syntactic correctness, this research emphasizes semantic fidelity, revealing a 30% gap between compilation and meaning preservation. The factorial analysis of tool interventions provides new insights into their distinct roles, highlighting elaboration feedback as the most impactful. The dataset and evaluation protocol establish a new benchmark for future research, bridging the gap between syntactic validity and semantic trustworthiness in AI formalization.
Limitations
- The reliance on GPT-5.2 and Gemini-2.5-Pro models for semantic judgment introduces biases and uncertainties, as these models may not fully capture mathematical nuances. Human calibration mitigates this but cannot eliminate model-specific errors.
- The dataset, although diverse, is limited to 400 statements, which may not fully represent the complexity and variety of mathematical language across all domains. Generalization to broader mathematical texts remains to be validated.
- Tool effects are analyzed in isolation, but in practice, interactions may be more complex, and some tool combinations could introduce unintended semantic shifts. Further refinement of tool integration strategies is needed.
Future Work
Future research will expand the dataset to include more complex and diverse statements, enhancing the robustness of the evaluation. Developing more sophisticated semantic judgment models, possibly incorporating symbolic reasoning, will improve accuracy. Further analysis of tool interactions and adaptive pipeline strategies will optimize the balance between validity and faithfulness. Exploring applications in automated theorem proving, educational tools, and collaborative mathematical research will accelerate the deployment of trustworthy AI in formal mathematics.
AI Executive Summary
This study addresses a fundamental challenge in automatic mathematical statement formalization: ensuring that generated formal statements are both syntactically valid and semantically faithful to the original natural language. Traditional benchmarks focus on proof success, but they overlook the core issue of meaning preservation. To tackle this, the authors constructed a curated dataset of 400 graduate-level mathematical statements spanning analysis, topology, and algebra. They developed a comprehensive evaluation protocol combining Lean compiler verification, semantic judgment by GPT-5.2 and Gemini-2.5-Pro models, and expert calibration, resulting in a high-precision faithfulness metric.
The key finding is that while the full system achieves a high compilation rate of 89.5%, the faithfulness rate drops to 60.5%, revealing a significant 30% gap where statements compile but do not preserve the intended meaning. Human audits confirm that most of these cases are genuine semantic mismatches, underscoring the insufficiency of compile success as a trust indicator. To understand the impact of various tool interventions, the authors employed a 23-factor factorial design, analyzing expert drafting, Mathlib search, and elaboration feedback.
Results show that elaboration feedback is the most effective for validity, significantly increasing the number of faithful statements, but also enlarging the semantic gap. Search tools improve grounding and selectivity, while expert drafting becomes largely redundant once feedback and grounding are available. Cross-model comparison demonstrates that the integrated tool-augmented system outperforms standalone models, achieving over 60% faithfulness.
Overall, this work advances the field by providing a scalable, high-precision evaluation framework that emphasizes semantic fidelity. It highlights the importance of multi-modal verification and tool synergy, guiding future efforts toward more trustworthy AI-driven formalization. Despite current limitations, including model biases and dataset scope, the methodology sets a new standard for evaluating and improving automatic mathematical statement formalization, with promising implications for AI-assisted mathematics research and education.
Deep Dive
Abstract
Theorem-proving benchmarks evaluate proof search against fixed formal statements, but natural-language-to-Lean formalization must generate the formal statement itself. In this setting, compilation is only a validity check: a Lean declaration may type-check while omitting hypotheses, changing domains, or expressing a vacuous claim. We study faithful statement formalization as both an evaluation problem and a bottleneck-attribution problem. On a 400-entry graduate-level benchmark spanning real analysis, complex analysis, topology, and algebra, our protocol combines Lean compilation, cross-model semantic judging, and human expert calibration. The resulting picture is different from compile-rate evaluation: a full tool-augmented agent reaches 89.5% compilation but only 60.5% consensus faithfulness, exposing a 29.0-point compile-pass but consensus-unfaithful gap. Targeted human audits support the metric as a conservative decision boundary: across available case-level audits, 96.0% of consensus-positive outputs are human-confirmed faithful, while 82.4% of compile-pass consensus-negative outputs are human-confirmed semantic failures. Under this metric, existing one-shot formalizer models and prover-oriented Lean models remain low, suggesting that formal validity, proof-oriented Lean competence, and faithful statement generation should be reported separately. We then use a full $2^3$ factorial design to decompose three recurring interventions in formalization pipelines: parametric expert drafting, Mathlib/context search, and Lean elaboration feedback. Elaboration feedback is the largest validity intervention, but it also exposes a larger compile-pass semantic-failure bucket; search mainly improves grounding and selectivity; and fine-tuned drafting is largely substitutable in this tool stack once feedback and grounding are available.