First Experiments with Neural Translation of Informal to Formal Mathematics

Mathematical formalization is essential for automated reasoning, verification, and knowledge sharing. Traditional rule-based systems like Coq, Mizar, and Flyspeck rely on manual encoding, which is labor-intensive and limited in scalability. Recent advances in deep learning, especially neural machine translation (NMT), have revolutionized natural language processing, inspiring efforts to apply similar techniques to mathematical texts. Prior works include probabilistic grammar models and semantic pruning, but these approaches struggle with scale and complexity. The scarcity of large aligned corpora has been a major bottleneck. This study leverages recent developments in seq2seq models, combined with large-scale corpus construction, to bridge the gap between informal LaTeX expressions and formal proof languages, aiming to automate the formalization process and facilitate AI-driven mathematical discovery.

The core challenge lies in translating informal, ambiguous, and symbol-rich mathematical expressions into precise, machine-verifiable formal language. Existing methods are constrained by limited datasets, leading to poor generalization and coverage. The complexity of mathematical notation, coupled with semantic ambiguities, makes purely rule-based or probabilistic approaches insufficient. Additionally, the lack of large, high-quality aligned corpora hampers deep learning applications. Addressing these issues requires innovative corpus construction, advanced neural architectures, and effective training strategies to achieve high accuracy and broad coverage in automatic formalization.

This work introduces several innovations: 1) constructing a large-scale aligned corpus of LATEX and Mizar expressions through automated informalization and alignment techniques, overcoming data scarcity; 2) applying Luong’s seq2seq neural translation architecture with multi-layer LSTM and scaled attention, optimized via hyperparameter tuning; 3) employing multi-model ensemble strategies to enhance coverage and robustness. Unlike prior rule-based or grammar-based methods, this approach leverages the generalization ability of deep neural networks to learn complex syntactic and semantic mappings, enabling scalable, high-accuracy translation of mathematical expressions. The pipeline integrates data preprocessing, tokenization, and hyperparameter optimization, setting a new benchmark for automated mathematical translation.

• �� Data collection: Use Bancerek’s LATEX-Mizar translation to generate formal-aligned pairs, resulting in about one million pairs.
• �� Data preprocessing: Tokenize LATEX and Mizar expressions, separate symbols, build vocabularies, and split datasets into training, validation, and testing.
• �� Model training: Implement Luong’s seq2seq with multi-layer LSTM, configure hyperparameters such as attention type (Scaled Luong), layer count (3 layers), and unit size (1024).
• �� Hyperparameter tuning: Use validation metrics (BLEU, perplexity) to optimize learning rate, batch size, residual connections, and optimizer (Adam preferred).
• �� Ensemble strategy: Combine multiple models with different hyperparameters, select top-performing models based on validation scores, and evaluate coverage and accuracy.
• �� Evaluation: Measure translation correctness using BLEU, exact match percentage, and edit distance metrics across test sets, analyze error cases, and compare configurations.

The experiments utilized a corpus of approximately 1,056,478 LATEX-Mizar pairs, split into training, validation, and test sets. Hyperparameters such as attention mechanism, layer depth, units, and optimizer were systematically varied. Performance was evaluated through BLEU scores, perplexity, and exact match percentages. Multiple configurations revealed that attention mechanisms and residual connections significantly improved translation quality. Ensemble models combining the top 4-6 configurations achieved coverage over 83%. The training process lasted around 11 hours on high-performance servers, demonstrating the approach’s scalability. The results confirmed neural networks’ capacity to learn complex symbolic mappings in mathematics, surpassing previous probabilistic grammar methods.

The optimal model, trained on 947,231 pairs for 11 hours, achieved 65.73% perfect translation accuracy on 105,247 test pairs, covering 79.17% with ensemble. Attention mechanisms, especially Scaled Luong, and three-layer LSTM units yielded the best performance. Ensemble strategies increased coverage, with the union of 39 models reaching 83.33%. The model effectively handled complex expressions, with many errors being semantically equivalent but syntactically different, indicating strong generalization. These results demonstrate the feasibility of neural translation in automating mathematical formalization at scale.

This technology can automate the conversion of informal mathematical texts into formal proof scripts, facilitating large-scale formalization projects, knowledge base construction, and automated theorem proving. It reduces manual effort, accelerates digitization of mathematical literature, and supports educational tools for formal reasoning. Future integration with semantic filtering and reasoning modules could enable fully autonomous formal proof generation, transforming mathematical research and education.

Despite promising results, the model struggles with highly ambiguous or intricate expressions, especially where semantic understanding is crucial. Training requires extensive computational resources and large annotated corpora, limiting accessibility. The approach primarily addresses syntactic correctness; semantic equivalence verification remains an open challenge. Further research is needed to incorporate reasoning and type-checking to ensure semantic fidelity, especially for complex or novel expressions.