Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering

Scientific question answering has evolved from simple fact retrieval to complex reasoning tasks. Early datasets like SQuAD and RACE focused on single-document comprehension, but struggled to emulate human reasoning. Recent advances include multi-hop datasets such as HotpotQA and MultiRC, which require chaining multiple facts. However, these often rely on structured knowledge bases or limited reasoning paths, lacking real-world scientific grounding. The advent of large pre-trained models like BERT improved performance but still fell short on tasks demanding deep understanding. Existing datasets rarely combine core scientific facts with broad commonsense, limiting their scope. OpenBookQA addresses this gap by integrating elementary science facts with everyday knowledge, providing a more realistic and challenging benchmark for AI reasoning.

Despite progress, current models are limited by knowledge retrieval bottlenecks and shallow reasoning capabilities. They often fail to effectively combine multiple knowledge sources or perform multi-hop inference necessary for complex scientific questions. This results in models that perform well on simple fact recall but poorly on questions requiring layered reasoning. The challenge is to develop systems that can efficiently retrieve relevant facts from large knowledge bases, understand their interrelations, and perform multi-step reasoning akin to human cognition. Addressing this problem is crucial for advancing AI's ability to understand and explain scientific concepts, which has broad implications for education, scientific discovery, and autonomous reasoning.

The key innovations include: 1) Constructing a new dataset, OpenBookQA, that combines core elementary science facts with broad commonsense knowledge, emphasizing multi-hop reasoning. 2) Developing a knowledge-aware neural model with attention mechanisms that fuse core facts and external knowledge dynamically, improving reasoning depth. 3) Designing oracle experiments to evaluate the theoretical upper bound of knowledge integration, revealing the importance of effective knowledge retrieval. 4) Implementing a multi-stage question generation pipeline that ensures question difficulty and quality, balancing challenge and answerability. These innovations collectively push the frontier of scientific QA by emphasizing knowledge fusion and multi-hop inference, providing both a benchmark and a methodological framework for future research.

• �� Start with a set of 1326 core facts (F) from the WorldTree corpus, filtered for elementary science relevance.
• �� Use crowdsourcing to generate questions by prompting workers to combine a core fact with additional common knowledge (k), forming a new assertion.
• �� Convert assertions into multiple-choice questions, ensuring answer plausibility and difficulty through automated and human filtering.
• �� Verify question hardness via IR and neural baselines, discarding questions answerable by simple methods.
• �� For each question, associate the core fact (f) and optional second facts (k) as auxiliary data.
• �� Develop a knowledge-aware neural model incorporating attention mechanisms to fuse facts and external knowledge sources like ConceptNet.
• �� Conduct oracle experiments by providing perfect knowledge to assess the potential performance ceiling.
• �� Evaluate models on train, dev, and test splits, analyzing accuracy, knowledge retrieval impact, and ablation effects.

The experimental setup involves training and testing models on the OpenBookQA dataset, with baselines including PMI, TableILP, TupleInference, and neural models like BiLSTM and knowledge-enhanced architectures. Metrics include accuracy and knowledge retrieval quality. Ablation studies examine the impact of core facts, external knowledge, and retrieval strategies. Oracle experiments simulate perfect knowledge retrieval to estimate upper bounds. Hyperparameters such as embedding size (300 GloVe vectors), model depth, and attention mechanisms are tuned. The dataset's difficulty is validated through human performance (~92%) and baseline comparisons, highlighting the challenge posed by multi-hop reasoning and knowledge integration.

Models without knowledge retrieval perform at 25%, while incorporating core facts and commonsense knowledge raises accuracy to 76%. Simple neural baselines reach only 48%, indicating bias and knowledge gaps. Oracle experiments show potential accuracy exceeds 80% with perfect knowledge, confirming the importance of retrieval. The results demonstrate that current models are limited by knowledge access and reasoning depth, and improvements in these areas could significantly boost performance. The findings also reveal that even with core facts, models struggle with multi-hop inference, emphasizing the need for better reasoning architectures.

The advancements in knowledge fusion and multi-hop reasoning can be applied to intelligent tutoring systems, scientific research assistants, and automated reasoning tools. These systems can provide more accurate explanations, support scientific discovery, and enhance educational tools by simulating human-like understanding. Integration with knowledge graphs and structured reasoning modules will further expand their capabilities, enabling real-time, explainable AI in complex domains.

Current models heavily depend on knowledge retrieval quality, which remains a bottleneck. The question generation process, though rigorous, may introduce biases and limit diversity. The models lack explicit reasoning structures, reducing their ability to handle complex inference paths. Computational costs for large-scale knowledge retrieval and reasoning are high. Future work must focus on scalable retrieval algorithms, structured reasoning frameworks, and end-to-end training to address these issues.