InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis
Proposes InfiniteScienceGym, a seed-based, verifiable scientific benchmark for evaluating reasoning and tool use, with models achieving only ~50% accuracy.
Key Findings
Methodology
The approach employs a deterministic simulator that, from a seed, constructs comprehensive scientific repositories, including directory structures, files, and tabular data. A privileged QA generator creates answerable and unanswerable questions with exact ground truth, ensuring verifiability. Questions are templated and paraphrased for robustness. Models interact with data via tool calls, enabling evidence-based reasoning without static datasets. Experiments show models reach only ~50% accuracy, with poor unanswerable question detection. Performance improves with increased tool interactions, emphasizing tool efficacy over token count.
Key Results
- Top model, GPT-5.4, achieves 49.2% accuracy; all models perform poorly on unanswerable questions. Tool calls significantly boost accuracy, indicating interaction is more effective than token volume. Error analysis reveals filtering and navigation as key failure points, with errors accumulating sequentially. Models often over-rely on context tokens, but stronger models prefer programmatic analysis, showing tool use is crucial.
- Model performance varies across question types; metadata and bivariate stats are easier, while directory traversal and univariate stats are hardest. Errors mainly occur in filtering and navigation steps, with performance improving as models progress further before error. Unanswerable question detection accuracy exceeds 80% for GPT-5.4 and Claude, but still leaves room for improvement.
- Increasing tool calls correlates with better accuracy, demonstrating the importance of interactive reasoning. Models often fail early in the reasoning chain, and token count alone does not reflect evidence processing. The benchmark exposes weaknesses in evidence grounding and uncertainty handling, guiding future model enhancements.
Significance
This work advances scientific AI evaluation by providing a controllable, reproducible, and scalable benchmark that overcomes biases inherent in real datasets. It enables precise assessment of models’ evidence-grounded reasoning, tool utilization, and uncertainty recognition. Such a platform is crucial for developing reliable AI assistants capable of autonomous scientific reasoning, hypothesis testing, and data analysis, addressing longstanding challenges in AI-driven scientific discovery.
Technical Contribution
The core innovation is a seed-based procedural generator that creates diverse, reproducible scientific repositories, combined with a templated and paraphrased QA system that guarantees verifiable ground truth. The integration of tool-mediated reasoning allows for detailed performance analysis in complex data environments. The framework supports unlimited scale without large static datasets, facilitating systematic evaluation of model capabilities in structured scientific tasks.
Novelty
This is the first benchmark to combine deterministic seed-based repository generation with verifiable, unanswerable question construction and natural language paraphrasing, enabling large-scale, reproducible evaluation of scientific reasoning. It uniquely assesses models’ evidence grounding, tool use, and uncertainty detection in a controlled environment, surpassing prior static dataset-based benchmarks.
Limitations
- While the simulated repositories are realistic, they cannot fully replicate the complexity of real-world scientific data, potentially limiting ecological validity.
- Models still struggle with identifying unanswerable questions, indicating the need for better uncertainty estimation and abstention strategies.
- The evaluation mainly focuses on structured data reasoning; extending to multi-modal data remains future work.
Future Work
Future directions include expanding the complexity and diversity of simulated scientific environments, integrating multi-modal data, and developing more sophisticated uncertainty estimation techniques. Enhancing models’ autonomous tool use and reasoning depth will further improve their applicability to real-world scientific tasks. Additionally, efforts to bridge simulated and real data scenarios will be crucial for practical deployment.
AI Executive Summary
In scientific research, evaluating AI models’ reasoning capabilities remains a significant challenge. Traditional benchmarks rely on published studies or human annotations, which are biased, noisy, and costly to distribute. To address these issues, Bentham and Srikumar introduce InfiniteScienceGym, a novel benchmark that leverages seed-based procedural generation to create unlimited, reproducible scientific repositories.
This platform comprises three main components: a simulator, a question-answer (QA) generator, and a paraphrasing module. The simulator, given a seed, constructs a complete scientific repository, including directory structures, files, and tabular data, ensuring reproducibility. The QA generator, with privileged access, produces answerable and unanswerable questions with guaranteed ground truth, covering a broad spectrum of scientific reasoning tasks. The paraphrasing module converts templated questions into natural language, making the evaluation more realistic.
Experiments with models such as GPT-5.4 and Claude Opus 4.6 reveal that the best accuracy achieved is only around 49.2%. The models perform poorly in detecting unanswerable questions, often providing confident but incorrect answers when evidence is lacking. The study highlights that increased tool interaction improves performance, emphasizing the importance of tool use over mere token volume. Error analysis shows filtering and navigation as key failure points, with errors accumulating sequentially.
This work significantly advances AI evaluation in scientific contexts by offering a scalable, bias-free, and verifiable benchmark. It opens pathways for developing models capable of evidence-grounded reasoning, uncertainty handling, and autonomous tool use, essential for reliable scientific AI assistants. Future efforts will focus on expanding environment complexity, multi-modal data integration, and enhancing model robustness for real-world deployment.
Deep Dive
Plain Language Accessible to non-experts
想象你在一个超级智能的图书馆里找资料。每次你来，图书馆会根据一个秘密的数字（就像游戏里的密码）自动整理出一套书架和资料。这些资料包括各种文件和表格，都是提前设计好的。你可以问它一些问题，比如“这个表格里最高的温度是多少？”，它会根据资料给你答案。如果资料里没有相关信息，它会告诉你“我不知道”。每次布置都由那个神秘数字决定，所以无论你什么时候来，资料都一样，方便验证答案的正确性。这个系统可以帮助我们测试人工智能在科学推理中的能力，特别是在面对没有足够证据的问题时，是否会勇敢地说“我不知道”或者错误地给出答案。
ELI14 Explained like you're 14
想象你在一个超级厉害的图书馆里找资料。每次你去，图书馆都会用一个神秘的密码（就像游戏里的密码）自动安排出一套书架和资料。这些资料包括各种表格和文件，都是提前设计好的。你可以问它一些问题，比如“这个表格里最高的温度是多少？”，它会根据资料告诉你答案。如果资料里没有相关信息，它会说“我不知道”。每次用的密码都不一样，所以每次都能测试这个图书馆是不是很聪明，能不能正确回答问题，或者知道自己不知道。这个系统就像一个超级智能的助手，帮你查资料，但它也会告诉你什么时候没有足够的证据，不能胡乱猜答案。
Abstract
Large language models are emerging as scientific assistants, but evaluating their ability to reason from empirical data remains challenging. Benchmarks derived from published studies and human annotations inherit publication bias, known-knowledge bias, label noise, and substantial storage requirements. We present InfiniteScienceGym, a procedurally generated benchmark of scientific repositories paired with a verifiable question-answering task. From a seed, the simulator deterministically generates a self-contained repository with realistic directory structure, files, and tabular data, and a privileged QA generator produces both answerable and unanswerable questions with exact ground truth. This makes it possible to evaluate evidence-grounded reasoning, abstention, and tool-mediated analysis in a controlled setting without distributing a large static corpus. InfiniteScienceGym complements real scientific benchmarks by targeting blind spots and failure modes that are hard to evaluate using published datasets alone. Evaluating both proprietary and open-weight models, we find that none achieve more than 50% accuracy overall, that recognizing unanswerable questions remains a major weakness, and that stronger models tend to use tools more effectively rather than simply consuming more tokens.