DRFLOW: A Deep Research Benchmark for Personalized Workflow Prediction
Proposed DRFLOW benchmark with 7 metrics, evaluating personalized workflow prediction across 100 tasks and 1246 steps, using multi-source evidence integration.
Key Findings
Methodology
This paper introduces DRFLOW, a comprehensive benchmark designed to evaluate personalized deep research workflows. The methodology involves a multi-stage data synthesis pipeline that generates realistic tasks grounded in five domains, utilizing company-side and user-side evidence. The core algorithm integrates knowledge graph-based structured modeling of workflows, multi-source retrieval mechanisms, and a set of seven diagnostic metrics covering factual grounding, step recovery, structural ordering, condition resolution, and personalization. The evaluation employs advanced deep learning models, including GPT-3.5 and Claude-Opus-4.5, optimized through multi-step reasoning and evidence fusion modules. The benchmark emphasizes not only the accuracy of action-step prediction but also the structural integrity and contextual personalization of the predicted workflows, providing a holistic assessment of model capabilities in complex, heterogeneous environments.
Key Results
- On five domains with 100 tasks, the proposed DRFA achieved an average F1 score improvement of 10.02% over strong baselines, with the highest scores reaching 85.99% in structural ordering and 69.43% in personalization metrics. The results demonstrate significant gains in factual grounding and step recovery, especially when multi-source evidence fusion mechanisms are employed. Ablation studies reveal that the integration of heterogeneous evidence and explicit structural modeling are critical for performance enhancement. The models consistently outperform baselines such as GPT-5.2 and DeepSeek-v3.2 across all metrics, with improvements of over 20% in key diagnostic measures, confirming the effectiveness of the proposed architecture.
- The experimental results validate the benchmark's ability to measure complex reasoning and personalized adaptation, showing that current models can substantially improve in structured, multi-source environments. The detailed analysis indicates that the proposed metrics effectively capture the nuanced aspects of workflow prediction, guiding future research toward more holistic and context-aware AI systems. The results also highlight the importance of multi-step reasoning modules and evidence grounding in achieving high-quality predictions in real-world scenarios.
- Practically, the benchmark and models developed provide a foundation for deploying intelligent assistants in enterprise knowledge management, automated process guidance, and decision support systems. The comprehensive evaluation framework ensures that future models can be systematically improved, addressing real-world complexities such as evidence noise, personalized conditions, and structural consistency.
Significance
This research marks a significant advancement in the evaluation of deep research systems, shifting focus from simple answer generation to structured, personalized workflow prediction. By integrating heterogeneous evidence sources and emphasizing structural and contextual accuracy, it addresses longstanding challenges in enterprise AI applications. The benchmark's multi-metric evaluation framework offers a nuanced understanding of model capabilities, fostering the development of more reliable, interpretable, and user-adaptive AI systems. Its potential impact spans academia, where it opens new avenues for research in structured reasoning, and industry, where it can transform knowledge workflows, automate complex procedures, and enhance decision-making processes. Overall, this work bridges the gap between theoretical AI capabilities and practical enterprise needs, paving the way for more intelligent, autonomous, and personalized AI assistants.
Technical Contribution
The paper's key technical contributions include the design of a multi-source, multi-stage data synthesis pipeline capable of generating realistic deep research tasks with ground-truth workflows grounded in heterogeneous artifacts. It introduces a comprehensive set of seven diagnostic metrics that evaluate factual grounding, structural integrity, and personalization, filling a critical gap in existing benchmarks. The development of DRFLOW-Agent (DRFA), which employs a multi-step reasoning framework combining evidence retrieval, structural modeling, and conditional inference, represents a significant step forward in structured workflow prediction. The integration of knowledge graph-based modeling with deep learning models such as GPT-3.5 and Claude-Opus-4.5 enables the system to handle complex reasoning tasks, adapt to personalized contexts, and produce structured, executable workflows. The benchmark's extensible pipeline facilitates domain adaptation and scalable task generation, supporting future research in diverse fields.
Novelty
This work is the first to establish a comprehensive benchmark for personalized, structured workflow prediction in deep research contexts, emphasizing multi-source evidence integration and structural reasoning. Unlike prior benchmarks focused on report or answer generation, DRFLOW evaluates the ability of models to predict actionable, personalized workflows grounded in heterogeneous artifacts. The introduction of seven diagnostic metrics provides a nuanced, multi-dimensional assessment framework, setting a new standard for evaluating complex reasoning and personalization in AI systems. The combination of a synthetic data pipeline, multi-step inference architecture, and detailed evaluation metrics constitutes a novel approach that advances the state-of-the-art in structured AI reasoning and enterprise automation.
Limitations
- Despite its comprehensive design, the current models still struggle with extreme cases involving high noise levels or conflicting evidence, indicating room for improvement in evidence filtering and conflict resolution mechanisms.
- The synthetic data generation pipeline, while realistic, may not fully capture the complexity and unpredictability of real-world enterprise environments, necessitating validation with real organizational data in future work.
- The computational complexity of multi-source retrieval and structured inference limits real-time deployment, especially in large-scale enterprise settings. Future work should focus on efficiency optimization and model compression.
Future Work
Future directions include incorporating real-world enterprise datasets to validate the generalization and robustness of模型，进一步提升模型在真实复杂场景中的表现。研究将探索更高效的多源信息融合算法，减少推理时间，提升系统的实用性。同时，将扩展指标体系，结合用户反馈机制，增强模型的个性化和解释能力。未来还将结合强化学习和自监督技术，优化工作流的连续性和可解释性，推动深度研究系统向更自主、更智能的方向发展。
AI Executive Summary
Deep research systems have become a cornerstone of advanced AI applications, aiming to automate complex, multi-step information gathering, reasoning, and knowledge synthesis tasks. However, existing evaluation frameworks predominantly focus on report or answer accuracy, neglecting the structured, actionable workflows that are crucial in enterprise contexts. Recognizing this gap, the present study introduces DRFLOW, a novel benchmark designed explicitly for personalized workflow prediction grounded in heterogeneous, multi-source evidence.
DRFLOW encompasses 100 meticulously synthesized tasks across five domains—B2B, B2C, Education, Healthcare, and Legal—each rooted in realistic deep research questions. These tasks simulate real-world scenarios where an AI agent must identify relevant evidence from scattered artifacts such as documents, emails, and chat logs, then predict a sequence of action steps that are both structurally coherent and personalized to the user's context. The data synthesis pipeline employs a multi-stage process: starting from task seeds, generating generic workflows based on company policies, deriving supporting insights and distractors, and finally customizing workflows with user-specific evidence. This pipeline ensures high fidelity and diversity, supporting scalable task generation.
To evaluate model performance comprehensively, the authors propose seven diagnostic metrics covering factual grounding, step recovery, structural ordering, condition resolution, and personalization. These metrics enable a nuanced assessment of the models' reasoning, structural integrity, and contextual adaptation. The core algorithmic framework, DRFLOW-Agent (DRFA), integrates multi-source evidence retrieval, knowledge graph-based workflow modeling, and multi-step reasoning modules, leveraging state-of-the-art large language models like GPT-3.5 and Claude-Opus-4.5.
Experimental results demonstrate that DRFA outperforms strong baselines, achieving an average F1 score increase of 10.02%. Notably, the system excels in structural ordering and personalization, with scores reaching 85.99% and 69.43%, respectively. These findings underscore the importance of multi-source evidence fusion and structured reasoning in complex workflow prediction. The benchmark's comprehensive evaluation framework provides a new standard for future research, emphasizing the importance of structural and personalized reasoning in AI systems.
Looking ahead, the authors plan to incorporate real enterprise data, optimize inference efficiency, and expand the metric set to include user feedback and explainability. These efforts aim to bridge the gap between research and practical deployment, fostering AI systems capable of autonomous, reliable, and personalized workflow management. Overall, this work significantly advances the state of the art, offering a rigorous, scalable, and realistic platform for developing next-generation deep research AI.
Deep Dive
Abstract
Deep research (DR) systems are increasingly used for complex information-seeking tasks, but existing works mainly focus on generating reports and summaries. In contrast, many enterprise tasks instead require an agent to identify concrete workflows which is a sequence of action-steps. For example, rather than summarizing budgeting policies, an agent should be able to determine the steps needed to answer a question such as: "How do I request new headcount given a fixed budget?". Therefore, we introduce DRFLOW, a benchmark for evaluating personalized workflows predicted by agents from heterogeneous sources. Each task requires the agent to identify relevant evidence from scattered sources, then use that evidence to predict the correct action-step sequence for the user's task. DRFLOW contains 100 tasks across five domains, with 1,246 reference workflow steps grounded in more than 3,900 sources. We define seven diagnostic metrics covering factual grounding, step recovery, structural ordering, condition resolution, and personalization. We further present DRFLOW-Agent (DRFA), a workflow-oriented reference agent to predict personalized workflow. We show that although DRFA improves over strong baseline agents (upto 10.02% average F1 score), there is substantial room for improvement remains across these workflow metrics, indicating that predicting complete and correct personalized workflows remains a challenging frontier for deep research.