STREET: A Multi-Task Structured Reasoning and Explanation Benchmark
STREET benchmark evaluates multi-task reasoning and explanation; GPT-3 and T5 models lag behind human performance.
Key Findings
Methodology
STREET benchmark includes tasks across domains like math, logic, and commonsense reasoning. Each question has a reasoning graph showing how premises lead to answers. Evaluated using GPT-3 and T5 models, focusing on generating structured reasoning steps.
Key Results
- GPT-3 performs well in math tasks with 34.8% accuracy but poorly in others.
- T5 excels in ARC and SCONE tasks with 93.5% and 69.6% accuracy, respectively.
- Generated reasoning graphs still lag behind human performance, especially in complex tasks.
Significance
STREET provides a unified framework for evaluating multi-step reasoning and explanation, filling a resource gap in NLP. It aids researchers in developing more robust models and improving model explainability.
Technical Contribution
Introduces reasoning graphs, structuring reasoning as directed acyclic graphs. Compared to existing datasets, STREET offers more complex reasoning structures and steps.
Novelty
STREET uniquely combines multi-task and multi-domain reasoning and explanation, significantly enhancing model explainability through reasoning graphs.
Limitations
- Models perform poorly in generating complex reasoning graphs, especially in AR-LSAT tasks.
- Current models' reasoning abilities are far below human levels.
Future Work
Future research could focus on improving models' ability to generate complex reasoning graphs and exploring better use of reasoning graphs in model training.
AI Executive Summary
The STREET benchmark introduces a multi-task, multi-domain natural language reasoning and explanation framework, addressing the inadequacy of existing QA datasets in evaluating reasoning capabilities. This framework requires models to not only answer questions but also generate structured reasoning steps showing how premises lead to conclusions. Evaluations of GPT-3 and T5 models reveal they lag behind human performance in generating reasoning graphs, especially in complex reasoning tasks. STREET offers researchers a new tool to develop more robust models and improve model explainability, advancing the field of NLP. However, current models still underperform in complex reasoning tasks, suggesting future research should focus on enhancing reasoning capabilities and explainability.
Deep Analysis
Background
The NLP field has long pursued endowing machines with reasoning capabilities. Traditional systems relied on symbolic or probabilistic knowledge for multi-step operations but struggled with ambiguity and learning reasoning rules. Recent advances in language models have improved direct reasoning over natural language, yet resources remain insufficient for comprehensive evaluation.
Core Problem
Existing QA datasets primarily focus on single-step reasoning, lacking evaluation of multi-step reasoning and explanation capabilities. Models need to generate structured reasoning steps to show how premises lead to answers.
Innovation
STREET innovatively combines multi-task and multi-domain reasoning and explanation, significantly enhancing model explainability through reasoning graphs. This framework fills a resource gap in NLP.
Methodology
- �� Tasks span domains like math, logic, and commonsense reasoning. • Each question includes a reasoning graph showing reasoning steps from premises. • Evaluated using GPT-3 and T5 models, focusing on structured reasoning step generation.
Experiments
Experimental design includes multiple datasets like GSM8K and AR-LSAT. Compared against baseline models, evaluating models' answer accuracy and reasoning graph generation capabilities. Experiments conducted with models of varying sizes to test performance across tasks.
Results
T5 excels in ARC and SCONE tasks with 93.5% and 69.6% accuracy, respectively. However, in generating reasoning graphs, models still lag far behind human performance, especially in complex reasoning tasks.
Applications
STREET can be used to evaluate and train more robust NLP reasoning models, particularly in applications requiring multi-step reasoning and explanation, such as education and legal fields.
Limitations & Outlook
Current models perform poorly in generating complex reasoning graphs, especially in AR-LSAT tasks. Models' reasoning abilities and explainability need improvement, suggesting future research should focus on these areas.
Plain Language Accessible to non-experts
Imagine you're in a kitchen following a recipe. The recipe tells you what ingredients you need and what steps to follow. STREET is like this recipe; it requires you to not only make a delicious dish (answer the question) but also explain how each step is completed (generate the reasoning graph). Just like in the kitchen, you need to know the reason and result of each step, and STREET requires models to show the complete reasoning process from the premises in the question.
ELI14 Explained like you're 14
Imagine you're playing a puzzle game. You need to find clues and solve the puzzle. STREET is like this game; it requires you to not only find the answer but also explain how you found it. Just like in the game, you need to reason step by step, and STREET requires models to show the complete reasoning process from the premises in the question. It's like showing your reasoning process in the game so others can understand your thinking!
Glossary
Reasoning Graph
A structured reasoning process showing how conclusions are derived from premises.
Used in STREET to show models' reasoning steps.
Multi-task Learning
A machine learning approach where a model learns multiple related tasks simultaneously.
STREET evaluates models' reasoning abilities across domains.
Explainability
The ability of a model to provide clear reasoning processes that humans can understand.
STREET requires models to generate structured reasoning steps.
GPT-3
A large language model capable of generating natural language text.
Used to evaluate reasoning capabilities in STREET.
T5 Model
A transformer-based language model skilled in generating and understanding natural language.
Used to evaluate reasoning capabilities in STREET.
Open Questions Unanswered questions from this research
- 1 How to improve models' performance in complex reasoning tasks, especially in generating reasoning graphs.
- 2 Current models' reasoning abilities and explainability need improvement, suggesting future research should focus on these areas.
Applications
Immediate Applications
Education
Helps develop more robust educational tools capable of explaining complex problem-solving processes.
Legal Field
Used for legal reasoning and decision support, providing clear reasoning processes.
Long-term Vision
General AI
Advances the development of general AI with stronger reasoning and explainability capabilities.
Abstract
We introduce STREET, a unified multi-task and multi-domain natural language reasoning and explanation benchmark. Unlike most existing question-answering (QA) datasets, we expect models to not only answer questions, but also produce step-by-step structured explanations describing how premises in the question are used to produce intermediate conclusions that can prove the correctness of a certain answer. We perform extensive evaluation with popular language models such as few-shot prompting GPT-3 and fine-tuned T5. We find that these models still lag behind human performance when producing such structured reasoning steps. We believe this work will provide a way for the community to better train and test systems on multi-step reasoning and explanations in natural language.