How well do Large Language Models perform in Arithmetic tasks?
Introduces MATH 401 dataset, evaluates GPT-4, ChatGPT on arithmetic tasks, showing GPT-4's superior performance with 83.54% accuracy.
Key Findings
Methodology
This study constructs the MATH 401 dataset, comprising diverse arithmetic operations—addition, subtraction, multiplication, division, exponentiation, trigonometry, and logarithms—covering integers, decimals, and irrational numbers. Using metrics such as accuracy, relative error, and non-number ratio, the evaluation assesses models including GPT-4, ChatGPT, InstructGPT, Galactica, and LLaMA across varying difficulties. The methodology involves multiple prompt strategies, including LaTeX formatting and instruction tuning, to analyze their impact. Experiments focus on long expressions, complex functions, and large numbers, systematically comparing model performance, scaling effects, and prompt sensitivities.
Key Results
- GPT-4 achieves the highest overall accuracy of 83.54%, demonstrating near-perfect performance on complex tasks such as large number multiplication, exponentiation, and logarithms, indicating advanced reasoning and numerical comprehension. ChatGPT follows with 75.06%. Model size, tokenization, and instruction tuning significantly influence results, with LATEX sources and prompt design enhancing performance. Chain-of-thought prompting surprisingly does not improve accuracy, revealing potential limitations in reasoning chains. GPT-4's ability to perform step-by-step calculations on long expressions underscores its superior inferencing capabilities.
- Analysis shows that larger models generally perform better, but beyond 30B parameters, gains plateau, suggesting a scale threshold. Tokenization strategies, especially digit-level splitting, impact arithmetic accuracy. Instruction tuning, especially reinforcement learning from human feedback (RLHF), markedly boosts performance. The experiments highlight that models like GPT-4 can handle complex, multi-step arithmetic with high precision, whereas smaller models struggle with high-difficulty tasks.
- Prompt design and input formatting are crucial; prompts like 'Calculate:' and LaTeX syntax yield the best results. The study also finds that models' performance varies across different operation types, with division, logarithms, and trigonometry being more challenging. The results provide a comprehensive benchmark for future development, emphasizing the importance of training data, model scale, and prompt engineering for enhancing arithmetic reasoning.
Significance
This research pioneers a systematic, high-density evaluation of large language models' arithmetic capabilities, establishing benchmarks that inform both academic research and practical applications. By quantifying the effects of model size, training corpus, and prompt strategies, it offers insights into the core factors influencing mathematical reasoning in AI. The findings demonstrate that models like GPT-4 possess near-human accuracy in fundamental arithmetic, paving the way for AI systems capable of supporting scientific discovery, automated theorem proving, and advanced mathematical education. The work also highlights the importance of specialized datasets in pushing the boundaries of AI reasoning abilities.
Technical Contribution
The paper introduces the MATH 401 dataset, a comprehensive benchmark covering multiple arithmetic operations at various difficulty levels. It employs a multi-metric evaluation framework, including accuracy, relative error, and non-number ratio, to provide a nuanced assessment. The study systematically analyzes the influence of tokenization (e.g., digit-level splitting), instruction tuning (including RLHF), and prompt engineering. It also compares models of different scales, revealing scale effects and thresholds. The work demonstrates that combining instruction tuning with high-quality datasets significantly enhances models' internal arithmetic reasoning, setting new standards for evaluating mathematical capabilities of LLMs.
Novelty
This is the first comprehensive, high-density benchmark explicitly targeting fundamental arithmetic abilities of large language models. Unlike prior work focused on word problems or symbolic math, MATH 401 emphasizes raw computational accuracy across diverse operations. The systematic analysis of prompt strategies, tokenization, and training methods provides novel insights into factors affecting arithmetic performance. The integration of multiple evaluation metrics and the focus on model scaling effects distinguish this work from existing benchmarks, offering a detailed understanding of the current limits and future directions for AI arithmetic reasoning.
Limitations
- The dataset mainly covers basic arithmetic and some functions, lacking in symbolic math, algebra, or higher-level calculus, limiting scope for advanced mathematical reasoning.
- Despite high accuracy, models still struggle with extremely large numbers, complex long expressions, and irrational constants, indicating room for improvement in numerical understanding.
- Experiments are constrained by API limits and model versions, and results may vary with future updates. Further research is needed to validate generalization across different tasks and modalities.
Future Work
Future directions include expanding the dataset to include symbolic algebra, calculus, and theorem proving tasks, enabling comprehensive evaluation of mathematical reasoning. Integrating external tools like symbolic solvers or calculators could enhance accuracy on complex problems. Investigating multi-modal inputs, such as formulas and diagrams, may improve understanding of mathematical concepts. Additionally, developing training strategies that combine supervised, reinforcement, and self-supervised learning will push models toward deeper mathematical comprehension and reasoning capabilities.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
Imagine a large factory where workers are responsible for different tasks—some assemble parts, others check quality, and一些负责计算成本。每个工人都需要按照说明书操作，确保每一步都正确。现在，科学家们就像是在测试这些工厂工人的数学能力。他们设计了一个特别的任务清单，让工人们用不同的方法计算各种数字，比如大数字乘法、复杂的三角函数和对数。结果显示，最新的“超级工人”——GPT-4，几乎没有出错，能像数学老师一样准确地完成任务。其他工人，比如ChatGPT，也表现不错，但还不能完全取代人类。这个研究帮助我们了解，未来的AI工人可以在科学、工程和学校中扮演更重要的角色，帮助人们解决复杂的数学问题。
ELI14 Explained like you're 14
想象你有一个超级聪明的机器人朋友，它每天帮你做数学作业。刚开始，它只会简单的加减乘除，但随着它变得更聪明，它可以解决更难的问题。科学家们也在测试这些机器人，看看它们能不能像人一样做数学题。最近，他们设计了一个特别的测试，让机器人做各种数学题，比如大数字的乘法、复杂的三角函数和对数。结果发现，最新的机器人——GPT-4，几乎没有出错，能像数学老师一样准确。其他机器人，比如ChatGPT，也表现不错，但还不够完美。这个研究很重要，因为它告诉我们，未来的AI可以帮助我们在科学、工程和学校学习中解决更难的问题，就像有了一个超级数学助手一样！
Abstract
Large language models have emerged abilities including chain-of-thought to answer math word problems step by step. Solving math word problems not only requires abilities to disassemble problems via chain-of-thought but also needs to calculate arithmetic expressions correctly for each step. To the best of our knowledge, there is no work to focus on evaluating the arithmetic ability of large language models. In this work, we propose an arithmetic dataset MATH 401 to test the latest large language models including GPT-4, ChatGPT, InstrctGPT, Galactica, and LLaMA with various arithmetic expressions and provide a detailed analysis of the ability of large language models. MATH 401 and evaluation codes are released at \url{https://github.com/GanjinZero/math401-llm}.