FinDeepIndicator: Benchmarking Deep Research Agents in End-to-End Financial Indicator Construction

Financial indicators are essential for translating raw financial data into actionable insights, evolving from manual calculations to automated systems powered by deep learning. Early models like Fama-French factors and CAPM laid the groundwork for quantitative finance. Recent advances involve neural networks and large language models, which promise automation but lack comprehensive evaluation frameworks. Existing benchmarks focus on answer correctness, neglecting the intermediate reasoning steps, which are crucial for trustworthiness. As AI models grow more complex, the need for process-aware evaluation becomes urgent to ensure reliability, interpretability, and practical utility in real-world financial decision-making. This study addresses this gap by proposing a structured, multi-stage assessment framework, aiming to foster more transparent and dependable financial AI systems.

Despite progress in financial modeling, current AI systems struggle with end-to-end indicator construction due to challenges in data retrieval, formula comprehension, and numerical accuracy. The lack of process-level evaluation hampers understanding of specific weaknesses, leading to unreliable outputs in critical applications like risk management and asset valuation. Moreover, macroeconomic indicators pose additional difficulties owing to heterogeneous data sources, reporting conventions, and complex reasoning requirements. This bottleneck limits the deployment of AI in real-time financial analysis, necessitating a systematic, fine-grained evaluation framework that can diagnose and guide improvements across all stages of indicator construction.

The key innovation lies in decomposing the indicator construction task into four distinct but interconnected stages, enabling detailed performance analysis. The framework employs a structured indicator taxonomy, covering fundamental, technical, and macroeconomic categories, with detailed metadata for each indicator. It introduces multi-modal validation, semantic equivalence scoring, and tolerance-aware numerical matching, ensuring robustness and interpretability. Automated question generation via template sampling ensures broad coverage and diversity, while the parsing and evaluation pipeline allows for precise diagnostics of model errors. This holistic, process-oriented approach surpasses traditional answer-only benchmarks, setting a new standard for evaluating financial AI systems.

• �� Indicator collection: Curate 234 indicators, define formulas, data needs, and metadata.
• �� Template design: Develop diverse templates with three difficulty levels, supporting multiple linguistic variants.
• �� QA generation: Instantiate templates with sampled variables, creating 3,350 QA pairs with detailed intermediate steps.
• �� Quality control: Use expert review and automated validation to ensure data and formula accuracy.
• �� Response parsing: Use LLMs to extract structured components—formulas, raw data, intermediate calculations, answers.
• �� Evaluation: Apply semantic matching, tolerance thresholds, and expert validation to assess correctness at each stage.
• �� Experiments: Test models on real market data, analyze performance across indicator types, markets, and difficulty levels.

The evaluation uses datasets from US and Chinese markets, covering 800 listed companies over ten years. Multiple models, including GPT-5, Claude, and specialized DR agents, are tested across various indicator categories. Metrics include accuracy in formula understanding, data retrieval, calculation, and final answers, with thresholds set at predefined tolerances. The experiments explore different task difficulties, market environments, and model configurations, providing a comprehensive performance landscape. Additional ablation studies analyze the impact of data quality, template complexity, and model size, validating the robustness of the evaluation framework.

Models achieve over 70% accuracy in formula understanding but only about 40% in final answers, highlighting bottlenecks in data and numerical accuracy. Deep research agents outperform search-based LLMs, especially in data retrieval and answer stages, with performance gaps of over 9% in US markets. Macro indicators remain the most challenging, with accuracy below 30%, mainly due to heterogeneous data sources and reasoning complexity. Cross-market comparisons reveal modest performance drops in Chinese markets, emphasizing data accessibility issues. These insights point to targeted areas for model improvement, especially in multi-source data integration and macroeconomic reasoning.

This benchmark enables systematic training and evaluation of financial AI models, improving their reliability in risk assessment, valuation, and macroeconomic forecasting. Financial institutions can use it to develop transparent, explainable tools that support regulatory compliance and decision-making. The framework also facilitates research on multi-modal data fusion, causal inference, and multi-step reasoning, accelerating innovation in automated financial analysis. Long-term, it aims to transform financial analysis from manual, heuristic-driven processes into fully automated, trustworthy AI-powered systems, reducing costs and increasing accuracy in financial services.

Models still struggle with macroeconomic indicators due to data heterogeneity and complex reasoning. The evaluation relies on predefined templates, limiting natural language variability and adaptability to unseen expressions. Cross-market generalization remains untested, and high computational costs hinder real-time deployment. Additionally, the current framework emphasizes structured indicators, which may not capture all nuances of real-world financial questions. Future work should focus on expanding indicator diversity, improving model robustness, and reducing computational overhead to enable broader practical adoption.