Augmenting Math Word Problems via Iterative Question Composing
Introduced IQC method to create MMIQC dataset; Qwen-72B-MMIQC achieves 45.0% accuracy on MATH benchmark.
Key Findings
Methodology
Proposed Iterative Question Composing (IQC), which uses GPT-4 to generate new questions and GPT-3.5 for rejection sampling, creating the MMIQC dataset to enhance mathematical reasoning.
Key Results
- Qwen-72B-MMIQC achieves 45.0% accuracy on MATH benchmark, surpassing open-source SOTA by 8.2% and outperforming GPT-4's initial version from 2023.
- MMIQC-enhanced models excel in Hungarian high school math exams, demonstrating strong generalization capabilities.
- Ablation studies show IQC contributes significantly, with standalone IQC data boosting model accuracy by 3.1%.
Significance
Addresses the performance bottleneck of open-source LLMs in solving complex math problems, offering a novel data augmentation method that advances both academic research and industry applications.
Technical Contribution
Combines high-quality pretraining corpora with synthetic data for fine-tuning. Introduces IQC to significantly enhance dataset diversity and mathematical reasoning performance.
Novelty
IQC iteratively generates complex problems, avoiding the limitations of traditional augmentation methods and significantly improving model performance.
Limitations
- IQC-generated problems may lack sufficient complexity to cover all mathematical domains.
- Rejection sampling relies on GPT-3.5's accuracy, potentially allowing some incorrect answers to pass.
- Fine-tuning costs are high, especially for large models like Qwen-72B.
Future Work
Future research could explore IQC's application in other domains, such as scientific question generation, and optimize sampling efficiency to reduce computational costs.
AI Executive Summary
Despite advancements in large language models (LLMs), solving competition-level math problems remains challenging for open-source models. This paper introduces a novel data augmentation method, Iterative Question Composing (IQC), which combines high-quality pretraining data and synthetic data to construct the MMIQC dataset. Fine-tuned models achieve remarkable performance on the MATH benchmark, with Qwen-72B-MMIQC reaching 45.0% accuracy, surpassing open-source SOTA and GPT-4's initial version.
IQC involves using GPT-4 to generate new questions and GPT-3.5 for rejection sampling, ensuring data quality and diversity. Experiments show IQC not only improves mathematical reasoning but also generalizes well to unseen problems, as demonstrated in Hungarian high school math exams.
However, IQC has limitations, such as the complexity of generated problems and high fine-tuning costs. Future work could explore its application in other fields and optimize efficiency to reduce costs further.
Deep Analysis
Background
Mathematical reasoning is a critical challenge in AI, particularly for competition-level problems. Open-source LLMs perform poorly on benchmarks like MATH compared to proprietary models like GPT-4 and Gemini-Ultra. Existing augmentation methods, such as rejection sampling and question bootstrapping, improve performance but face limitations in data diversity and quality.
Core Problem
Open-source LLMs struggle with complex math problems due to limited data quality and reasoning capabilities. Building high-quality, diverse datasets to enhance model performance remains a pressing challenge.
Innovation
Introduced IQC, which iteratively generates complex problems to expand datasets. Combined high-quality pretraining corpora with synthetic data for fine-tuning. IQC avoids traditional augmentation limitations, significantly improving data diversity and model performance.
Methodology
- �� Used GPT-4 to generate new questions based on seed problems.
- �� Applied GPT-3.5 for rejection sampling to filter incorrect answers.
- �� Constructed MMIQC dataset with 1.2M question-answer pairs.
- �� Fine-tuned multiple models, including Qwen-72B, to optimize mathematical reasoning.
Experiments
Evaluated models on MATH benchmark using zero-shot settings to extract answers and compare accuracy. Tested generalization on Hungarian high school math exams. Conducted ablation studies to analyze contributions of MMIQC subsets.
Results
Qwen-72B-MMIQC achieves 45.0% accuracy on MATH, outperforming MetaMathQA fine-tuned models by 8.2%. IQC alone boosts model accuracy by 3.1%. Hungarian exam results validate generalization capabilities.
Applications
Applicable in math education for automated problem generation and solving assistance. Useful for training competitive math teams and advancing research in mathematical reasoning.
Limitations & Outlook
IQC-generated problems may lack sufficient complexity to cover all domains. Fine-tuning costs are high, especially for large models. Rejection sampling depends on GPT-3.5's accuracy, potentially allowing some errors.
Plain Language Accessible to non-experts
Imagine you’re running a factory that produces puzzles. IQC is like a smart machine that takes a basic puzzle and keeps making new, more challenging versions of it. It also has a quality checker to ensure every puzzle works perfectly before shipping.
ELI14 Explained like you're 14
Think of IQC like a super-smart friend in a video game who takes the puzzles you’ve solved and creates even cooler ones for you to try! Plus, they make sure every new puzzle has a fair solution so you can keep leveling up!
Glossary
IQC (Iterative Question Composing)
A data augmentation method that iteratively generates complex problems to expand datasets.
Used to create synthetic problems in the MMIQC dataset.
MMIQC
A dataset containing high-quality math problems and answers for fine-tuning LLMs.
Improves performance on MATH benchmark.
Rejection Sampling
A method to filter incorrect answers and ensure data quality.
Used in IQC to validate generated questions.
MATH Benchmark
A test set of 5,000 competition-level math problems.
Evaluates mathematical reasoning capabilities of models.
Qwen-72B
An open-source large language model fine-tuned for mathematical reasoning.
Achieved state-of-the-art results on MATH benchmark.
Open Questions Unanswered questions from this research
- 1 How to further enhance IQC-generated problem complexity to cover more domains?
- 2 How to optimize rejection sampling efficiency to reduce computational costs?
- 3 What are the broader applications of IQC in other fields?
Applications
Immediate Applications
Math Education
Automated problem generation and solving assistance for students learning complex topics.
Competition Training
Provides high-quality training problems for math competition participants.
Long-term Vision
Cross-Domain Problem Generation
Applying IQC to fields like science and logical reasoning to advance AI applications in complex problem-solving.
Abstract
Despite the advancements in large language models (LLMs) for mathematical reasoning, solving competition-level math problems remains a significant challenge, especially for open-source LLMs without external tools. We introduce the MMIQC dataset, comprising a mixture of processed web data and synthetic question-response pairs, aimed at enhancing the mathematical reasoning capabilities of base language models. Models fine-tuned on MMIQC consistently surpass their counterparts in performance on the MATH benchmark across various model sizes. Notably, Qwen-72B-MMIQC achieves a 45.0% accuracy, exceeding the previous open-source state-of-the-art by 8.2% and outperforming the initial version GPT-4 released in 2023. Extensive evaluation results on Hungarian high school finals suggest that such improvement can generalize to unseen data. Our ablation study on MMIQC reveals that a large part of the improvement can be attributed to our novel augmentation method, Iterative Question Composing (IQC), which involves iteratively composing new questions from seed problems using an LLM and applying rejection sampling through another LLM.