FinProBench: Evaluating Financial AI Agents with Role-Grounded Rubrics Derived from Professional Deliverables

The financial industry increasingly relies on AI for tasks like report generation, risk analysis, and compliance. Early research focused on NLP tasks such as sentiment analysis and named entity recognition (Shah et al., 2022; Lei et al., 2023), progressing to reasoning and question-answering benchmarks (Islam et al., 2023). Recent models like BloombergGPT (Wu et al., 2023) and FinGPT (Yang et al., 2023) demonstrate strong performance on isolated tasks. However, these benchmarks primarily evaluate factual recall or reasoning within limited scopes, neglecting the complexity of professional deliverables such as multi-page reports, valuation models, and regulatory analyses. The industry demands evaluation frameworks that can assess the quality, coherence, and compliance of long-form, structured outputs, which existing benchmarks fail to address. This gap limits the ability to reliably gauge AI readiness for real-world financial applications, impeding trust and adoption.

Current evaluation methods for financial AI are inadequate for measuring performance on authentic professional deliverables. They often rely on prompt-based or output-only assessments, which do not reflect the tacit standards and nuanced quality criteria used by practitioners. Manual creation of standards is costly and inconsistent, while existing automated methods lack grounding in real work products. This results in evaluations that are superficial, non-transferable across roles, and insufficient for industry deployment. The core challenge is to develop a scalable, objective, and role-specific evaluation framework that captures the implicit standards embedded in authentic deliverables, enabling meaningful assessment of AI systems in real-world financial tasks.

This paper introduces RGRC, a novel, four-stage pipeline that automates the extraction of tacit professional standards from authentic deliverables. Unlike prior methods relying on model outputs or prompts, RGRC leverages real-world documents to identify role-specific competencies, synthesize structured rubrics, and validate their discriminative power through multi-evaluator mechanisms. The approach emphasizes role-based standard reuse, hierarchical rubric design, and automated quality assurance, significantly reducing manual effort and enhancing transferability. This innovation bridges the gap between superficial prompt-based evaluation and authentic professional standards, enabling scalable assessment across diverse financial roles and tasks.

• �� Deliverable Collection: Curate ≥20 authentic documents per role from authoritative sources, ensuring diversity and authenticity.
• �� Competency Extraction: Use LLMs to analyze each deliverable, identifying structural patterns, reasoning depth, domain-specific conventions, and quality indicators.
• �� Cross-Document Synthesis: Aggregate competencies appearing in ≥60% of deliverables to form role-level standards, distinguishing universal, role-specific, and implicit standards.
• �� Rubric Synthesis: Convert competencies into structured, scored criteria following principles of evaluability, unambiguity, non-hackability, and orthogonality, with complexity-adaptive criteria count.
• �� Validation: Apply automated checks for coverage, alignment, and consistency; conduct discriminative review with multiple evaluators; iterate until κ ≥ 0.75.
• �� Role-to-task transfer: Inherit 60-70% of criteria from role standards, adding task-specific criteria for evaluation efficiency.

The evaluation involved 20 tasks across 20 roles, with datasets comprising authentic deliverables from industry sources. Four heterogeneous LLM judges assessed deliverables using role-level rubrics, with multiple scoring rounds to measure agreement (κ=0.76). Metrics included coverage rate, ranking accuracy, and consistency. Baseline comparisons with prompt-only methods demonstrated RGRC’s superior coverage (up to 99.1%) and discriminative ability. Ablation studies confirmed the importance of role-based standards and multi-evaluator validation. Results showed human deliverables outperformed AI systems, but the framework reliably distinguished quality differences across models and tasks.

RGRC achieved 99.1% coverage in specialized roles, far exceeding prompt-only methods (78%). Human deliverables scored an average of 73.7, with AI systems around 70, confirming evaluation reliability. Role-level rubric reuse reduced construction effort by 6.7 times, enabling scalable deployment. The multi-model, multi-evaluator design ensured consistent, objective assessments, validating the framework’s robustness across diverse scenarios.

This benchmark can be directly used for evaluating financial report generation, valuation models, and compliance analyses. It supports industry adoption by providing a scalable, objective, and professional standard for AI performance assessment, fostering trust and regulatory compliance. Long-term, the framework can evolve into an industry-wide standard, enabling continuous improvement of AI systems and supporting regulatory audits.

The approach depends on publicly available, high-quality data, which may not cover all niche roles or regions. Authenticity verification remains challenging at scale. Automated scoring, while robust, still benefits from manual oversight to prevent bias. Future work should focus on expanding data sources, refining validation, and integrating adaptive learning to improve accuracy and coverage.