A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers
ASDiv dataset enhances math word problem solver evaluation accuracy through diverse lexicon and problem types.
Key Findings
Methodology
The ASDiv dataset includes 2,305 English math word problems covering most types taught in elementary schools. Each problem is annotated with its type and grade level. A lexicon usage diversity metric is proposed to assess dataset diversity.
Key Results
- ASDiv outperforms existing datasets in diversity metrics, showing a more accurate reflection of solver capabilities.
- Experiments show that current state-of-the-art systems perform significantly below human levels on ASDiv, with accuracy around 36%.
- Comparing different grade levels, accuracy significantly drops for higher-grade problems.
Significance
ASDiv addresses the limitations of existing datasets by enhancing diversity in problem types and lexicon usage, making AI system evaluations more akin to real human test scenarios, thus advancing AI applications in education.
Technical Contribution
ASDiv makes significant contributions to dataset diversity, providing a wider range of problem types with consistent annotations, supporting more precise system evaluations. The proposed lexicon diversity metric offers a new perspective for dataset construction.
Novelty
ASDiv is the first dataset to emphasize both problem type and lexicon diversity, significantly improving the accuracy of solver capability evaluations.
Limitations
- The dataset's relatively small size may not cover all potential problem types.
- Some complex problems still require additional domain knowledge to solve.
Future Work
Future work includes expanding the dataset size, adding more complex problem types, and developing smarter solvers to tackle diverse problems.
AI Executive Summary
The ASDiv dataset enhances the evaluation accuracy of math word problem solvers through diverse lexicon usage and problem types. Existing datasets often have limitations in language patterns or problem types, leading to overly optimistic AI system evaluations. ASDiv addresses this by covering most problem types taught in elementary schools and annotating each problem with its type and grade level.
Experimental results show that current state-of-the-art solvers perform significantly below human levels on ASDiv, with accuracy around 36%. This indicates that ASDiv can more accurately reflect solver capabilities, advancing AI applications in education.
Despite ASDiv's significant contributions to dataset diversity, its relatively small size necessitates further expansion to cover more potential problem types. Future work will focus on expanding the dataset and developing smarter solvers.
Deep Analysis
Background
Math word problems are crucial for evaluating AI's natural language understanding capabilities. Existing datasets often have limitations in language patterns or problem types, leading to overly optimistic AI system evaluations. ASDiv addresses this by increasing diversity.
Core Problem
Existing datasets lack diversity and consistent annotations, leading to inaccurate AI system evaluations. A more diverse and consistently annotated math word problem dataset is needed.
Innovation
ASDiv's innovation lies in its diverse lexicon usage and problem type coverage. By proposing a lexicon diversity metric, ASDiv can more accurately assess solver capabilities.
Methodology
- �� Collected 2,305 English math word problems
- �� Annotated problem type and grade level
- �� Proposed lexicon usage diversity metric
- �� Conducted experiments to validate dataset diversity and solver evaluation accuracy
Experiments
Experiments used the ASDiv dataset to evaluate existing solvers' performance, showing significantly lower accuracy than human levels. Comparing different grade levels revealed a significant drop in accuracy for higher-grade problems.
Results
ASDiv outperforms existing datasets in diversity metrics, showing a more accurate reflection of solver capabilities. Current state-of-the-art systems perform significantly below human levels on ASDiv, with accuracy around 36%.
Applications
The ASDiv dataset can be used to evaluate and develop smarter math word problem solvers, particularly in education, helping improve students' math skills.
Limitations & Outlook
Despite ASDiv's significant contributions to diversity, its relatively small size necessitates further expansion to cover more potential problem types.
Plain Language Accessible to non-experts
Imagine you're shopping in a large supermarket; ASDiv is like this supermarket, offering a wide variety of products (i.e., math problems). Existing datasets are like small stores with only a few products. ASDiv provides a more diverse selection, allowing you to more comprehensively test your shopping skills (i.e., solving skills).
ELI14 Explained like you're 14
Imagine you're playing a math game, and ASDiv is like a super level with various challenges. Existing datasets are like simple beginner levels with few challenges. ASDiv lets you test your math skills more comprehensively, boosting your game level!
Glossary
ASDiv
ASDiv is a diverse English math word problem dataset designed to evaluate and develop solvers.
Used to assess AI systems' true capabilities.
Lexicon Usage Diversity
A metric measuring the diversity of vocabulary and language patterns in a dataset.
Used to assess dataset diversity.
Math Word Problem
A natural language description of a math problem requiring understanding and reasoning to solve.
Used to test AI's natural language understanding capabilities.
Elementary Education
Refers to math problem types taught at the elementary school level.
Problem types covered by the ASDiv dataset.
Solver
An AI system designed to solve math word problems.
Assessing AI system capabilities.
Open Questions Unanswered questions from this research
- 1 How to further increase dataset size and diversity to cover more potential problem types.
- 2 How to develop smarter solvers to tackle the diverse problems in the ASDiv dataset.
Applications
Immediate Applications
Educational Assessment
ASDiv can be used to assess students' math skills, helping teachers develop more effective teaching plans.
Long-term Vision
Intelligent Education Systems
Developing intelligent education systems based on ASDiv to provide personalized learning experiences.
Abstract
We present ASDiv (Academia Sinica Diverse MWP Dataset), a diverse (in terms of both language patterns and problem types) English math word problem (MWP) corpus for evaluating the capability of various MWP solvers. Existing MWP corpora for studying AI progress remain limited either in language usage patterns or in problem types. We thus present a new English MWP corpus with 2,305 MWPs that cover more text patterns and most problem types taught in elementary school. Each MWP is annotated with its problem type and grade level (for indicating the level of difficulty). Furthermore, we propose a metric to measure the lexicon usage diversity of a given MWP corpus, and demonstrate that ASDiv is more diverse than existing corpora. Experiments show that our proposed corpus reflects the true capability of MWP solvers more faithfully.