DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?
Introduces DSAgentBench, a benchmark for evaluating AI agents' ability to automate full data science workflows in real OS environments, achieving only 56.7% success.
Key Findings
Methodology
This study develops DSAgentBench, comprising 275 complex, multi-stage tasks covering data acquisition, exploration, modeling, visualization, and validation within a real Ubuntu OS environment. Tasks are designed through expert analysis and collaboration with large language models (LLMs), ensuring realistic data science workflows. Deterministic evaluators verify correctness, visualization, and model performance, surpassing simple code execution metrics. Experiments evaluate 15 models, including closed-source (Claude-4.6-Sonnet) and open-source variants, focusing on multi-tool coordination and long-horizon reasoning. The environment supports multimodal observations (screenshots, UI trees) and GUI actions, simulating real-world scenarios. Results show Claude-4.6-Sonnet achieves 56.7% success, while open-source models remain below 1%, exposing significant gaps in system understanding and tool orchestration.
Key Results
- Claude-4.6-Sonnet outperforms open-source models with a success rate of 56.7%, yet overall performance remains far from human levels, highlighting the difficulty of end-to-end automation.
- Model performance varies significantly across task complexity and tool usage, with frequent failures in environment grounding, multi-step reasoning, and tool coordination, especially in complex, multi-tool workflows.
- Experiments reveal that current models struggle with system-level understanding and managing intermediate states, emphasizing the need for more integrated reasoning and environment comprehension modules.
Significance
This work is the first comprehensive benchmark assessing AI agents’ capacity to autonomously execute complete data science workflows within real operating systems. It exposes the substantial gap between current AI capabilities and the demands of practical, multi-tool, multi-stage data analysis. The results underscore the importance of developing systems with deeper system understanding, better tool orchestration, and long-horizon reasoning. The benchmark serves as a foundation for future research aimed at creating reliable, autonomous data science agents, with potential impacts spanning industry automation, decision-making, and AI-assisted research. It addresses a critical bottleneck in deploying AI for real-world data analysis tasks, pushing the field toward more robust, verifiable autonomous systems.
Technical Contribution
The paper introduces a novel benchmark integrating real OS interaction with complex, multi-stage data science tasks. It combines multimodal observations (screenshots, UI trees) with GUI-based actions, enabling agents to perform end-to-end workflows involving data retrieval, cleaning, modeling, and visualization. The deterministic evaluation framework ensures precise correctness verification beyond code execution, assessing analytical validity and model performance. The environment supports diverse tools like Jupyter, VS Code, and databases, fostering realistic scenarios. The comprehensive evaluation of 15 models reveals critical limitations in current systems, guiding future research directions in system understanding, tool coordination, and long-horizon reasoning, thus advancing the development of autonomous, verifiable data science agents.
Novelty
This is the first benchmark to evaluate autonomous agents executing full data science workflows within a real operating system environment, emphasizing tool coordination, environment grounding, and multi-step reasoning. Unlike prior static code correctness benchmarks, DSAgentBench simulates real-world workflows with diverse, multi-stage tasks, integrating multimodal observations and GUI interactions. Its comprehensive design bridges the gap between isolated skill evaluation and end-to-end automation, setting a new standard for assessing AI's practical capabilities in complex, real-world data analysis scenarios.
Limitations
- Despite improvements, models still struggle with complex multi-tool coordination and system-level understanding, leading to frequent failures in environment manipulation and intermediate state management.
- The environment is currently limited to Ubuntu OS, which may restrict generalizability to other platforms like Windows or macOS; future work should extend compatibility.
- Evaluation focuses primarily on success rate, lacking in-depth analysis of efficiency, robustness, and interpretability, which are crucial for industrial deployment.
Future Work
Future research will focus on enhancing models’ system comprehension and long-horizon reasoning, integrating reinforcement learning and meta-learning techniques. Expanding environment support to multiple OS platforms and increasing task diversity will improve robustness and generalization. Additionally, efforts will target optimizing interaction efficiency, explainability, and scalability, aiming to develop fully autonomous, reliable data science agents capable of operating in complex, real-world settings.
AI Executive Summary
In today’s data-driven world, automating the entire data science workflow remains a significant challenge. Traditional benchmarks have primarily focused on isolated tasks such as code correctness or simple data manipulations, failing to capture the complexity of real-world data analysis. Recognizing this gap, the authors introduce DSAgentBench, a comprehensive benchmark designed to evaluate AI agents’ ability to autonomously perform end-to-end data science tasks within a real operating system environment.
This benchmark encompasses 275 multi-stage, multi-tool tasks that mirror actual data science workflows—from data acquisition and cleaning to modeling, visualization, and validation. Built on a realistic Ubuntu environment, DSAgentBench supports multimodal observations (screenshots, UI trees) and GUI actions, enabling agents to interact with multiple tools such as Jupyter, VS Code, and databases. Tasks are carefully curated through expert analysis and collaboration with large language models, ensuring they reflect practical analytical challenges.
Experimental results reveal that even the most advanced models, like Claude-4.6-Sonnet, achieve only 56.7% success across tasks. All open-source models perform below 1%, exposing critical limitations in system understanding, tool coordination, and long-horizon reasoning. These findings highlight the substantial gap between current AI capabilities and the demands of real-world data science workflows. The study emphasizes the need for deeper system comprehension and integrated reasoning modules.
Overall, DSAgentBench sets a new standard for evaluating autonomous data science agents, fostering research toward more reliable, explainable, and scalable AI systems. Its development marks a pivotal step toward industrial automation of complex analytical tasks, promising transformative impacts on data-driven decision-making in various sectors. Future directions include expanding environment support, enhancing model robustness, and integrating reinforcement learning to achieve truly autonomous, verifiable data science workflows.
Deep Analysis
Background
Data science has evolved from manual scripting to automation with tools like R, Python, and specialized platforms. Early benchmarks such as HumanEval focused on code correctness, but lacked real-world complexity. Recent efforts like DS-1000 and MLAgentBench introduced more nuanced tasks, yet remained limited to static code evaluation or isolated stages. The advent of large language models (LLMs) like GPT-4 and Claude has spurred interest in AI-assisted data analysis, but these systems primarily generate code snippets or manage task context without full environment interaction. Industry reports highlight persistent challenges: AI models struggle with tool orchestration, environment grounding, and multi-step reasoning in real workflows. This backdrop motivated the creation of DSAgentBench, aiming to evaluate AI agents' end-to-end capabilities in realistic settings, bridging the gap between isolated skill testing and practical automation.
Core Problem
The core challenge lies in enabling AI agents to autonomously execute comprehensive data science workflows within real operating systems. Existing models excel at isolated tasks like code generation but falter in coordinating multiple tools, managing intermediate states, and understanding system environments. This limits their deployment in real-world scenarios where data scientists interact with diverse applications—databases, notebooks, IDEs—requiring continuous reasoning and environment manipulation. The problem is compounded by the lack of benchmarks that evaluate full workflows with environment interaction, making it difficult to measure progress toward autonomous, reliable AI data scientists. Addressing this gap is crucial for industrial automation, reducing manual effort, and improving decision-making efficiency.
Innovation
This work introduces several key innovations: 1) a comprehensive, real OS-based benchmark with 275 multi-stage tasks covering the entire data science lifecycle; 2) integration of multimodal observations (screenshots, UI trees) and GUI-based actions, enabling realistic environment interaction; 3) deterministic evaluation metrics that verify analytical correctness, visualization quality, and model performance, moving beyond static code tests; 4) support for multiple tools including Jupyter, VS Code, and databases, simulating real workflows. These innovations collectively enable a more accurate assessment of AI agents’ capabilities in complex, multi-tool scenarios, fostering advancements in autonomous system development and practical deployment.
Methodology
- �� Environment setup: Ubuntu OS with pre-installed Python, pandas, scikit-learn, supporting multimodal observation (screenshots, UI trees) and GUI actions.
- �� Task design: Based on expert analysis and LLM collaboration, 275 tasks span data acquisition, exploration, feature engineering, modeling, and validation, reflecting real-world workflows.
- �� Task configuration: Each task includes a system state (datasets, tools), natural language instructions, and a Python evaluator.
- �� Interaction loop: Agents perceive via screenshots/UI trees, perform GUI actions (click, drag, keyboard), and execute up to 15 steps per task.
- �� Evaluation: Deterministic scripts verify output correctness, visualization fidelity, and model metrics, ensuring comprehensive assessment.
- �� Experiments: Multiple models evaluated, including GPT-4o, GPT-5, Claude-4.6-Sonnet, and open-source variants, across varied task complexities and tool combinations.
Experiments
The evaluation involved 15 models, spanning proprietary (GPT-4o, GPT-5, Claude-4.6-Sonnet) and open-source architectures (UI-TARS, GUI-OWL). Each model attempted all 275 tasks, with success rate as primary metric. Tasks varied in complexity (easy to hard), data modality (tabular, image, text), and tool usage (single or multiple). Hyperparameters were tuned for fairness, and ablation studies assessed the impact of different observation inputs. The environment recorded detailed logs of actions, states, and outcomes, enabling in-depth analysis of failure modes. Results demonstrated that Claude-4.6-Sonnet achieved 56.7% success, while open-source models hovered below 1%, highlighting the gap in system understanding and multi-step reasoning. Additional experiments tested robustness across task difficulty and tool complexity.
Results
Claude-4.6-Sonnet significantly outperformed open-source models, with a success rate of 56.7%, yet still far from human performance (~85%). Most open-source models failed to surpass 1%, often in tool orchestration and environment grounding. Tasks involving multiple tools and stages revealed critical weaknesses in environment understanding and intermediate state management. Ablation studies showed that multimodal observations (screenshots + UI trees) improved performance by approximately 10%, but models still struggled with multi-step planning. The results underscore the necessity for models to develop deeper system comprehension and reasoning capabilities to handle complex workflows reliably.
Applications
This benchmark enables development of autonomous data analysis systems capable of managing end-to-end workflows in real environments. Industries such as finance, healthcare, and research can leverage such agents for automated report generation, model deployment, and continuous data monitoring, reducing manual effort and error. Educational institutions can use it for training students in practical data science skills. Long-term, this work paves the way for fully autonomous AI-driven data pipelines, transforming how organizations handle large-scale data analysis and decision-making, ultimately leading to smarter, faster, and more reliable AI-assisted workflows.
Limitations & Outlook
Current models still lack robust environment understanding, often failing at tool coordination and intermediate state management. The environment is limited to Ubuntu, reducing cross-platform applicability. The evaluation primarily measures success rate, neglecting efficiency, interpretability, and robustness metrics. Computational costs remain high for large models, limiting scalability. Future work must address these issues by improving system comprehension, expanding environment support, and developing more efficient, explainable models to realize practical autonomous data science agents.
Plain Language Accessible to non-experts
想象你在一家厨房做饭，这个厨房里有很多不同的厨具：锅、刀、调料瓶、烤箱等等。你要做一道复杂的菜，比如一锅炖菜。你首先得买材料（数据），然后洗菜、切菜（探索分析），接着放到锅里炖（建模），最后摆盘（可视化）并尝尝味道（验证）。每一步都需要用不同的厨具，不能只会做最后的菜，还要会用各种厨具、知道什么时候放调料、什么时候调火。这就像让一个机器人助手学会在厨房里自己操作所有厨具，完成一道复杂的菜。这项研究就是在教这个助手如何在真实厨房里自己动手，从买材料到做出一道美味的菜，整个流程都能自己完成。
Abstract
Real-world data science involves long-horizon workflows that span data wrangling, exploration, modeling, visualization, and validation, and require coordinated use of tools such as notebooks, IDEs, terminals, browsers, and databases within real operating environments. Yet existing benchmarks lack real-computer interaction and do not evaluate whether agents can execute complete end-to-end data-science workflows in realistic computing environments, failing to capture the multi-stage, multi-tool nature of data-science practice. We introduce DSAgentBench, the first benchmark to evaluate whether agents can automate full data-science workflows inside real computer environments. DSAgentBench contains 275 diverse tasks covering the entire data-science life-cycle, reflecting the complexity and tool coordination required in practice. Each task requires grounding decisions in intermediate outputs and coordinated tool use, and includes a deterministic evaluator that verifies analytical correctness, visual outputs, and model performance rather than code-only execution. Our extensive experiments with 15 closed- and open-source models show that even the strongest agent, Claude-4.6-Sonnet, achieves only 56.70% task success, while all open-source agents remain below 1%, frequently failing at tool orchestration, OS grounding, and multi-step reasoning. These results reveal a substantial capability gap between current agentic systems and real data-science workflows, positioning DSAgentBench as a foundation for developing grounded, verifiable, autonomous data-science agents. We release DSAgentBench at https://github.com/vis-nlp/DSAgentBench.