MathCoPilot: An Interactive System for Human-AI Symbiotic Paradigm of Mathematical Research
MathCoPilot integrates human-in-the-loop proof blueprinting, multi-skill orchestration, and automated knowledge base construction, achieving 20/63 success on FormalMATH with natural language prior to formalization.
Key Findings
Methodology
The system combines an interactive proof blueprint, adaptive skill orchestration, and automated knowledge base management. The proof blueprint allows researchers to steer high-level proof design while AI handles detailed formalization. Multiple proving strategies (NL-first, formalization-first, knowledge-grounded) are dynamically scheduled based on intermediate feedback, with Lean verification ensuring correctness. Experiments compare four models (Gemini 3.1 Pro, GPT-5.4, Claude Opus 4.7, Claude Sonnet 4) across FormalMATH subset and two PDE theorems, analyzing success rates, error detection, and recursive refinement capabilities, demonstrating the effectiveness of human-AI collaboration in complex proof tasks.
Key Results
- In FormalMATH, models with natural language prior outperform pure formalization, with success rates of 20/63 (~31.75%) versus 10/63 (~15.87%), a 15.87% improvement. In PDE theorems, models achieve 100% verification under well-formed formalizations, with recursive refinement boosting solutions for previously unsolved cases. The knowledge base’s automatic extraction and maintenance significantly enhance model knowledge utilization, supporting personalized research workflows. The system’s dynamic skill switching and interactive blueprints improve proof success, especially in complex scenarios.
- Models excel in undergraduate-level problems, notably in algebra and calculus, but struggle with advanced analysis and non-standard libraries. Error detection improves with natural language scaffolding, yet subtle logical errors remain challenging. The recursive refinement process effectively solves some difficult proofs, validating the human-AI synergy. The knowledge base’s continuous update mechanism ensures relevance and accuracy, fostering an evolving research environment. Overall, the system demonstrates promising potential for automating research-level mathematics while highlighting areas for further enhancement.
- The adaptive orchestration of multiple proof strategies, combined with an interactive proof blueprint, allows the system to handle diverse proof scenarios. Researchers can guide high-level proof design while AI manages low-level formalization, reducing trial-and-error. The experimental results confirm that human-in-the-loop workflows significantly outperform fully automated approaches in complex proofs, especially when leveraging natural language prior. This work paves the way for more intelligent, flexible mathematical assistants capable of supporting advanced research tasks.
Significance
This research advances the frontier of automated mathematical reasoning by integrating human intuition with AI automation. It addresses longstanding challenges in formal proof automation, such as handling complex, domain-specific theorems and maintaining knowledge consistency. The system’s modular design fosters scalable, personalized research environments, bridging the gap between theoretical AI capabilities and practical mathematical research needs. Its innovative human-in-the-loop approach sets a new standard for intelligent scientific tools, promising to accelerate discovery, improve verification reliability, and democratize access to advanced mathematical reasoning. The framework also offers a blueprint for integrating AI into other scientific domains requiring complex reasoning and formal verification.
Technical Contribution
The core technical innovation lies in the 'proof blueprint' framework, enabling real-time human guidance and step-wise formalization. The multi-skill orchestrator employs reinforcement learning-inspired strategies to adaptively select proof tactics, improving efficiency and success rates. The automatic knowledge base construction leverages large language models to extract, formalize, and verify definitions and theorems from literature, creating a continuously evolving repository. The integration with Lean verification ensures correctness at every step, while the recursive refinement mechanism allows tackling previously unsolvable proofs. These advances collectively push the boundary of AI-assisted formal mathematics, providing a flexible, scalable, and user-centric proof environment.
Novelty
This work is the first to embed a dynamic, human-guided proof blueprint within an AI system that orchestrates multiple proof skills and maintains an evolving knowledge base. Unlike prior fully automated systems, it emphasizes researcher control and interpretability, enabling high-level guidance and real-time adjustments. The recursive refinement approach for complex proofs and the integrated knowledge extraction pipeline represent significant departures from existing methods, which often rely on static pipelines or single-strategy approaches. This paradigm shift fosters more reliable, transparent, and adaptable AI-assisted mathematical research.
Limitations
- The system’s performance diminishes on highly complex, non-standard, or poorly formalized problems due to limited training data and知识库覆盖范围。
- 错误检测能力仍需提升，尤其在微妙逻辑错误和非线性偏微分方程等高阶问题中表现不足。
- 在处理极大规模证明时存在计算瓶颈，未来需优化推理效率和知识库扩展策略。
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂里有很多不同的机器，每台机器负责不同的任务。有的机器擅长组装，有的擅长检测质量。MathCoPilot就像是一个智能的工厂管理系统，它能帮你规划整个生产流程。你告诉它你想做的产品，它会帮你设计一份详细的生产蓝图，拆分成每个步骤。你可以随时调整某个步骤，系统会根据你的反馈，自动调度不同的机器（模型）来完成任务。它还能从大量的生产资料中自动提取相关信息，存入知识库，方便以后使用。整个过程就像你在管理一个高效、智能的工厂，既能控制大局，又能让机器帮你完成繁琐的细节。这样一来，复杂的生产任务变得简单多了，效率也大大提升。
ELI14 Explained like you're 14
想象你在学校里做一个大项目，里面有很多步骤，比如写计划、做实验、整理资料、写报告。你可以请一个聪明的助手帮你，但你得告诉他你的想法，他再帮你把细节搞定。MathCoPilot就像这个助手，它能帮你规划整个项目，把大任务拆成小步骤，你可以随时告诉它要改什么。它还会从书里、网上找相关资料，存到自己的资料库里，方便以后用。它会用不同的方法帮你完成每个步骤，比如先用简单的方法试试，再用复杂的技巧解决难题。这样，你就可以专注于想做的事情，助手帮你处理繁琐的细节，项目变得更快更顺利。它让复杂的数学证明变得像做学校作业一样简单有趣。
Abstract
Existing LLM-based theorem provers have achieved impressive results on formal mathematics benchmarks, yet they remain confined to acting as autonomous agents that prove a stated proposition. In this paper, we propose MathCoPilot, a human-in-the-loop system that embodies a new human--AI symbiotic paradigm for mathematical research, in which the mathematician steers the high-level mathematical direction while AI agents carry out the detailed formalization and proof work under continuous human guidance. MathCoPilot unifies three core capabilities: (1) an interactive workbench where the mathematician and AI agents collaborate through a living proof blueprint that decomposes a proof into navigable steps the human can directly inspect, direct, and refine; (2) automated proving skill orchestration with adaptive knowledge base search and Lean-integrated iterative verification; and (3) topic-driven paper retrieval and automated formalization into a verified Lean knowledge base. Using MathCoPilot, we systematically compare four state-of-the-art LLMs, including Gemini~3.1~Pro, GPT-5.4, and Claude~Opus~4.7, on a FormalMATH subset and on two real PDE theorems requiring deep domain expertise, evaluating their ability to produce verified Lean~4 proofs and to identify errors in deliberately incorrect proofs. Our results show that while current models can handle undergraduate-level problems with high success rates under favorable autoformalization conditions, substantial challenges remain for domain-specific theorems requiring genuine mathematical understanding.