Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correction
Goedel-Prover-V2 achieves new breakthroughs in automated theorem proving with scaffolded data synthesis and self-correction, reaching 88.1% accuracy on MiniF2F.
Key Findings
Methodology
Goedel-Prover-V2 employs innovations like scaffolded data synthesis, verifier-guided self-correction, and model averaging. Scaffolded data synthesis generates synthetic tasks of increasing difficulty to enhance the model's ability to master complex theorems. Verifier-guided self-correction leverages Lean compiler feedback to iteratively revise proofs. Model averaging merges model checkpoints to mitigate the decline in output diversity during later training stages.
Key Results
- Goedel-Prover-V2-8B outperforms DeepSeek-Prover-V2-671B on MiniF2F with 84.6% pass@32, despite being 80 times smaller.
- The flagship model Goedel-Prover-V2-32B achieves 88.1% pass@32 on MiniF2F, reaching 90.4% in self-correction mode.
- On PutnamBench, Goedel-Prover-V2-32B solves 86 problems at pass@184, significantly surpassing DeepSeek-Prover-V2-671B's 47 problems.
Significance
Goedel-Prover-V2 sets a new benchmark in open-source theorem proving, especially under constrained computational resources. It not only leads in benchmarks like MiniF2F and PutnamBench but also offers an efficient solution for academia and industry. The success demonstrates the potential for high performance without massive models and computational resources.
Technical Contribution
The study introduces scaffolded data synthesis and verifier-guided self-correction, significantly enhancing theorem proving efficiency and accuracy. Compared to existing SOTA methods, Goedel-Prover-V2 achieves higher performance with smaller model size and compute budget. The application of model averaging effectively addresses the diversity issue in later training stages.
Novelty
Goedel-Prover-V2 is the first to integrate verifier feedback with long-chain reasoning, forming an efficient self-correction mechanism. This approach excels in complex reasoning tasks, offering higher accuracy and efficiency compared to previous theorem proving methods.
Limitations
- The model may still make errors on complex theorems, especially with insufficient training data.
- The self-correction process can increase reasoning time, affecting real-time applications.
- The model's generalization ability in specific domains needs further validation.
Future Work
Future research directions include optimizing the self-correction process to reduce reasoning time, expanding the model's application to more mathematical domains, and exploring more efficient data synthesis methods to further enhance model performance.
AI Executive Summary
Goedel-Prover-V2 is a series of open-source language models that have made significant advancements in automated theorem proving through innovative methods. Existing solutions typically rely on large-scale models or computationally intensive inference, but Goedel-Prover-V2 improves performance while significantly reducing computational demands through methods like scaffolded data synthesis and self-correction.
The core technologies of the model include scaffolded data synthesis, verifier-guided self-correction, and model averaging. Scaffolded data synthesis helps the model master complex theorems by creating synthetic tasks of increasing difficulty. Verifier-guided self-correction uses feedback from the Lean compiler to iteratively revise proofs, significantly improving model accuracy. Model averaging maintains output diversity by merging model checkpoints.
Experimental results show that Goedel-Prover-V2 performs excellently on benchmarks like MiniF2F and PutnamBench, especially under constrained computational resources. The model not only sets a new performance benchmark in open-source theorem proving but also provides an efficient solution for academia and industry. However, the model's performance on certain complex theorems still needs improvement, and future research will continue to optimize these aspects.
Deep Analysis
Background
Automated theorem proving is a major challenge in AI, requiring the construction of machine-verifiable formal proofs. Recent years have seen significant advancements with systems like DeepMind's AlphaProof and AlphaGeometry achieving IMO-level performance. However, these successes typically depend on massive models or computationally intensive inference.
Core Problem
Automated theorem proving requires constructing rigorous logical flows in formal languages, posing a significant challenge for AI systems. Existing methods often rely on large-scale models or computationally intensive inference, making efficient theorem proving difficult under constrained computational resources.
Innovation
Goedel-Prover-V2 introduces innovations like scaffolded data synthesis, verifier-guided self-correction, and model averaging, significantly enhancing theorem proving efficiency and accuracy. Scaffolded data synthesis generates synthetic tasks of increasing difficulty to enhance the model's ability to master complex theorems. Verifier-guided self-correction leverages Lean compiler feedback to iteratively revise proofs. Model averaging merges model checkpoints to mitigate the decline in output diversity during later training stages.
Methodology
- �� Scaffolded Data Synthesis: Generates synthetic tasks of increasing difficulty to help the model master complex theorems.
- �� Verifier-guided Self-correction: Uses Lean compiler feedback to iteratively revise proofs.
- �� Model Averaging: Merges model checkpoints to maintain output diversity.
- �� Reinforcement Learning: Optimizes the model's reasoning ability through a multi-task setup.
Experiments
Experiments were conducted on benchmarks like MiniF2F and PutnamBench, using pass@N as the main evaluation metric. The model was tested under different inference budgets to verify its performance under constrained computational resources. Experiments also included ablation studies on self-correction and model averaging to assess their impact on model performance.
Results
Goedel-Prover-V2 achieves 88.1% pass@32 on MiniF2F, surpassing DeepSeek-Prover-V2-671B, despite being 80 times smaller. On PutnamBench, Goedel-Prover-V2-32B solves 86 problems at pass@184, significantly surpassing DeepSeek-Prover-V2-671B's 47 problems.
Applications
Goedel-Prover-V2 has broad applications in mathematical competitions, academic research, and industrial applications. Its efficient theorem proving capabilities can be used to solve complex mathematical problems and support mathematical education and research.
Limitations & Outlook
Despite its excellent performance on multiple benchmarks, Goedel-Prover-V2's performance on certain complex theorems still needs improvement. Additionally, the self-correction process can increase reasoning time, affecting real-time applications. Future research will continue to optimize these aspects.
Plain Language Accessible to non-experts
Imagine a smart student learning math. They start with simple problems and gradually tackle more complex ones. This is like Goedel-Prover-V2's scaffolded data synthesis method, which trains the model by generating tasks of increasing difficulty. Then, when the student makes mistakes in an exam, the teacher gives feedback, and they correct their answers based on this feedback. This is similar to the verifier-guided self-correction process, where the model uses Lean compiler feedback to revise proofs. Finally, the student summarizes all their learning experiences, forming a complete knowledge system, much like model averaging, which maintains output diversity by merging model checkpoints.
ELI14 Explained like you're 14
Imagine you're playing a super complex game with many levels, each harder than the last. Goedel-Prover-V2 is like a super smart gamer, getting better by playing these levels. Whenever it fails a level, the game tells it what went wrong, and it adjusts its strategy based on these hints. This is like its self-correction feature. In the end, it sums up all its experiences and becomes even stronger. Isn't that cool?
Glossary
Automated Theorem Proving
Using computer programs to automatically generate formal proofs of mathematical theorems.
Used in the paper to evaluate Goedel-Prover-V2's performance.
Scaffolded Data Synthesis
Generating synthetic tasks of increasing difficulty to train the model.
Used to enhance the model's ability to master complex theorems.
Verifier-guided Self-correction
Using compiler feedback to revise the model's proofs.
Used to improve model accuracy.
Model Averaging
Merging model checkpoints to maintain output diversity.
Used to address the decline in diversity during later training stages.
MiniF2F
A benchmark for evaluating automated theorem proving models.
Used to assess Goedel-Prover-V2's performance.
Open Questions Unanswered questions from this research
- 1 How to improve the model's reasoning ability on complex theorems with insufficient training data?
- 2 How to optimize the self-correction process to reduce reasoning time?
- 3 How to enhance the model's generalization ability in specific domains?
Applications
Immediate Applications
Mathematical Competitions
Goedel-Prover-V2 can be used to solve complex problems in mathematical competitions, helping participants improve their problem-solving skills.
Long-term Vision
Mathematical Education
By providing efficient theorem proving tools, Goedel-Prover-V2 can support mathematical education and research, promoting the development of the mathematical field.
Abstract
We introduce Goedel-Prover-V2, a series of open-source language models that set a new state-of-the-art in automated theorem proving. Built on the standard expert iteration and reinforcement learning pipeline, our approach incorporates three key innovations: (1) Scaffolded data synthesis: We generate synthetic tasks of increasing difficulty to train the model to master increasingly complex theorems; (2) Verifier-guided self-correction: We enable the model to iteratively revise its proofs by leveraging feedback from the Lean compiler; (3) Model averaging: We merge model checkpoints to mitigate the decrease in model output diversity in later stages of training. Our small model, Goedel-Prover-V2-8B, reaches 84.6% pass@32 on MiniF2F and outperforms DeepSeek-Prover-V2-671B under the same metric, despite being 80X smaller. Our flagship model, Goedel-Prover-V2-32B, achieves 88.1% on MiniF2F at pass@32 in standard mode and 90.4% in self-correction mode, outperforming prior SOTA by a large margin. Additionally, our flagship model solves 86 problems on PutnamBench at pass@184, securing the first place among open-source models on the leaderboard, surpassing DeepSeek-Prover-V2-671B's record of solving 47 problems by pass@1024 with a significantly smaller model size and compute budget. At the time of its release (July-August 2025), Goedel-Prover-V2 achieves the strongest overall performance among all open-source theorem provers. It also ranks among the top-performing models--including closed-source systems with publicly reported performance--under a constrained test-time compute budget. Our models, code, and data are released at https://github.com/Goedel-LM/Goedel-Prover-V2.