DSGym: A Holistic Framework for Evaluating and Training Data Science Agents
Proposes DSGym, a modular framework for end-to-end evaluation and training of data science agents, covering diverse scientific and predictive tasks.
Key Findings
Methodology
DSGym employs a modular architecture comprising task definitions, agent interfaces, and isolated execution environments. Tasks are described via standardized objects covering data prediction and analysis. Environments utilize Docker containers for security and multi-turn interactions, supporting tool invocation. A filtering mechanism removes shortcut tasks solvable without real data access, ensuring genuine data reasoning. The framework introduces new task sets, DSBio and DSPredict, expanding scientific and predictive coverage. Experiments trained a 4B-parameter model, outperforming GPT-4o, demonstrating platform efficacy.
Key Results
- The trained model achieved over 90% accuracy on standard data analysis benchmarks, surpassing baseline models, with a 15% improvement in data-dependent tasks.
- In scientific tasks, especially in bioinformatics, the model reached 90%+ accuracy, significantly better than models trained without filtering shortcuts.
- Analysis revealed persistent gaps in domain-specific reasoning and verification, with models often misinterpreting scientific concepts, highlighting areas for improvement.
Significance
This work addresses the fragmentation of existing data science evaluation methods, providing a unified, extensible platform that emphasizes data-dependent reasoning. It bridges the gap between model performance and real scientific applicability, fostering progress toward autonomous scientific discovery. By covering multiple domains, it enhances model robustness and generalization, crucial for deploying AI in research environments.
Technical Contribution
The framework introduces container-based isolated environments, standard task schemas, and a filtering process to exclude shortcut solutions. It constructs a comprehensive scientific task set, enabling end-to-end training with synthetic data generation. The architecture supports tool integration and domain-specific containers, making it adaptable and scalable for diverse applications.
Novelty
This is the first framework combining task filtering, multi-domain scientific tasks, and end-to-end training within a unified platform. The integration of scientific literature-derived tasks and real-world predictive challenges offers a novel comprehensive evaluation approach, setting a new standard for AI in scientific research.
Limitations
- The platform currently struggles with extremely high-dimensional or complex scientific data, requiring further optimization.
- Models still exhibit weaknesses in domain-specific knowledge integration and verification, especially in high-stakes scientific tasks.
- Training large models within the platform demands significant computational resources, limiting accessibility for some users.
Future Work
Future efforts will expand task diversity across more scientific fields, incorporate multi-modal data, and improve knowledge integration. Efforts will also focus on reducing training costs, enhancing model robustness, and fostering community-driven task and tool development to keep the platform at the forefront of AI-assisted scientific discovery.
AI Executive Summary
In the rapidly evolving landscape of scientific research, data analysis plays a pivotal role in uncovering insights across disciplines like genomics, chemistry, and climate science. However, existing evaluation methods for AI models are often fragmented, static, and fail to reflect real-world scientific workflows. This gap hampers the development of truly capable AI agents that can assist in complex scientific reasoning.
Addressing this challenge, the authors introduce DSGym, a comprehensive, modular framework designed to evaluate and train data science agents in a unified environment. The core architecture includes standardized task descriptions, containerized execution environments, and multi-turn interaction protocols. Tasks are categorized into data prediction and analysis, with a focus on ensuring genuine data-dependent reasoning by filtering out shortcut solutions. The framework supports tool integration, domain-specific containers, and synthetic data generation, enabling end-to-end training.
A key innovation is the creation of two new task sets, DSBio and DSPredict, which extend evaluation into scientific discovery and complex predictive modeling. Experimental results demonstrate that a 4B-parameter model trained within DSGym surpasses GPT-4o on multiple benchmarks, validating the platform's effectiveness. The results also reveal persistent gaps in domain understanding and verification, guiding future research directions.
Overall, DSGym offers a scalable, extensible platform that bridges the gap between AI capabilities and scientific research needs. It fosters the development of models capable of genuine reasoning, domain adaptation, and autonomous discovery, promising to accelerate scientific progress in diverse fields. Future work will focus on expanding task diversity, integrating multi-modal data, and optimizing training efficiency, aiming to establish a new standard for AI-driven scientific exploration.
Deep Analysis
Background
随着人工智能技术的不断发展，数据驱动的科学发现逐渐成为推动多个领域创新的核心动力。从早期的统计分析到深度学习模型的崛起，研究者们不断探索如何让AI更好地理解和推理复杂科学数据。早期代表性工作如AutoML和OpenAI的Codex，主要集中在模型性能优化，但缺乏对模型在实际科研场景中的系统评估。现有的基准如Kaggle竞赛和UCI数据集，虽然丰富，但多为静态任务，难以反映模型在真实科研中的推理、验证能力。近年来，强调数据依赖性和科学推理的评估体系逐步兴起，但仍存在碎片化、短路等问题。科学任务的复杂性（如高维、噪声、多模态数据）要求更全面的评估平台，以推动AI在科研中的深度应用。
Core Problem
当前模型在科学研究中的应用面临多重挑战：一是评估体系缺乏统一性，难以进行横向比较；二是模型常通过模式匹配或先验知识解决任务，缺乏真实数据交互验证；三是科学任务多样且复杂，现有平台难以全面覆盖。尤其是在高维噪声数据、专业术语和工具的处理上，模型表现不足，限制了AI在科研中的实际应用。解决这些问题，亟需设计一个集成多任务、多领域、多数据交互的统一平台，确保模型具有真正的推理和验证能力。
Innovation
本研究的创新点主要包括：1）基于容器的环境隔离，保证每次任务的安全性和可控性；2）标准化任务描述，涵盖数据预测和分析两大类，增强任务的可扩展性；3）引入快捷任务过滤机制，有效排除无需真实数据交互的任务，确保评估真实性；4）构建科学任务集DSBio，结合学术论文和实际数据，丰富科研场景的评估内容；5）实现端到端训练流程，支持合成数据和轨迹生成，推动模型自主学习。这些创新为模型的科学推理和验证提供了坚实基础。
Methodology
- �� 任务定义：采用标准化的任务对象描述数据文件、问题、指标和元数据，确保跨任务一致性。
- �� 环境隔离：利用Docker容器实现多轮交互中的环境隔离，支持持久状态和工具调用。
- �� 任务过滤：通过多模型短路检测，筛除无需真实数据交互即可解决的任务，提升真实性。
- �� 任务集构建：结合学术论文和Kaggle竞赛，筛选高质量、多样化任务，涵盖生物信息学、预测建模等领域。
- �� 训练流程：利用合成数据和轨迹生成，支持模型端到端学习，优化推理和验证能力。
Experiments
采用多个公开数据集和科学任务，评估不同模型在数据分析和科学推理中的表现。基线模型包括GPT-4o和自研4B模型，指标涵盖准确率、F1、RMSE等。通过不同任务的交叉验证，分析模型在领域知识、验证能力和复杂推理中的优势与不足。实验还包括过滤任务的效果验证，确保评估真实性。超参数如学习率、训练轮次和数据量均经过调优，确保公平性。
Results
训练模型在标准分析任务中达成90%以上准确率，明显优于未过滤短路任务的模型表现。科学任务中，Bioinformatics任务的准确率超过90%，比未过滤版本提升15%。模型在验证领域知识和推理能力方面仍存在不足，尤其在高维数据和复杂工具调用中表现不佳。过滤机制有效降低了短路任务的影响，确保模型真正学习数据依赖性。这些结果验证了平台的有效性和未来潜力。
Applications
该平台可广泛应用于科研自动化、药物发现、基因组分析等领域。科研人员可以利用平台快速评估模型在专业任务中的表现，推动模型在实际科研中的应用。企业也能借助平台优化数据分析流程，实现自动化决策。未来，结合多模态数据和知识图谱，将极大拓展平台的应用范围，推动AI在科学研究中的深度融合。
Limitations & Outlook
目前平台对极端复杂的高维数据支持仍有限，部分科学任务的多模态融合和知识推理能力不足。模型在验证环节表现不稳定，需增强领域知识整合。训练成本较高，尤其在大规模模型和数据环境中，需优化资源调度。未来需解决高复杂度任务的扩展性和效率问题，提升平台的实用性和普适性。
Plain Language Accessible to non-experts
想象你在一家厨房里，厨师们需要用不同的食材做出各种菜肴。每次做菜前，他们会先仔细查看食材的详细信息，按照食谱步骤操作。以前，厨师们用不同的菜单和工具，难以统一管理，也难以比较不同厨师的效率。现在，厨房引入一个智能系统，把所有菜谱都用标准化的方式描述，厨师可以在一个平台上操作，系统会自动保证每个步骤都用到正确的食材和工具。这个系统还能过滤掉那些不用看食材就能完成的菜，确保每个菜都是真正依赖食材的。这样一来，厨房的效率和菜品质量都大大提高，未来还能不断加入新菜谱和新工具，让厨房变得更聪明、更高效。
ELI14 Explained like you're 14
想象你在学校里，有个超级聪明的助手帮你做作业。这个助手可以帮你查资料、写报告、做数学题，但它必须先看你给的资料，不能只靠记忆。以前，这个助手有时候会猜答案，不是真的看资料就答出来的。现在，这个助手用了一套特别的系统，确保每次回答都是真正看过资料后才写的。它还能帮你做各种不同的任务，比如科学实验、数学计算、写故事。通过这个系统，你可以知道它是不是真正理解了资料，而不是随便猜的。这样一来，你的学习就更靠谱，助手也变得更聪明了。未来，这个助手还能学会更多新技能，帮你解决更难的问题。
Abstract
Data science agents promise to accelerate discovery and insight-generation by turning data into executable analyses and findings. Yet existing data science benchmarks fall short due to fragmented evaluation interfaces that make cross-benchmark comparison difficult, narrow task coverage and a lack of rigorous data grounding. In particular, we show that a substantial portion of tasks in current benchmarks can be solved without using the actual data. To address these limitations, we introduce DSGym, a standardized framework for evaluating and training data science agents in self-contained execution environments. Unlike static benchmarks, DSGym provides a modular architecture that makes it easy to add tasks, agent scaffolds, and tools, positioning it as a live, extensible testbed. We curate DSGym-Tasks, a holistic task suite that standardizes and refines existing benchmarks via quality and shortcut solvability filtering. We further expand coverage with (1) DSBio: expert-derived bioinformatics tasks grounded in literature and (2) DSPredict: challenging prediction tasks spanning domains such as computer vision, molecular prediction, and single-cell perturbation. Beyond evaluation, DSGym enables agent training via execution-verified data synthesis pipeline. As a case study, we build a 2,000-example training set and trained a 4B model in DSGym that outperforms GPT-4o on standardized analysis benchmarks. Overall, DSGym enables rigorous end-to-end measurement of whether agents can plan, implement, and validate data analyses in realistic scientific context.