FinQA: A Dataset of Numerical Reasoning over Financial Data
Introduces FinQA, a financial QA dataset with expert-annotated multi-step numerical reasoning, highlighting model performance gaps.
Key Findings
Methodology
FinQA employs a retrieval-generation framework, leveraging pre-trained models like BERT and RoBERTa to first retrieve relevant facts from financial reports, then generate executable reasoning programs composed of domain-specific operations. Data is annotated by financial experts, including question-answer pairs and reasoning steps, ensuring transparency. The models progressively build multi-step programs, involving operations such as addition, subtraction, multiplication, division, and table aggregation. Experiments show the best model achieves 65.05% execution accuracy, still far below experts at 91.16%, indicating room for improvement in financial reasoning capabilities.
Key Results
- FinQA contains 8,281 question-answer pairs, with an average of 2.7 reasoning steps per program. Operations predominantly involve division and addition, reflecting typical financial calculations. The RoBERTa-large model achieves 58.86% in program accuracy, outperforming non-expert human performance (87.49%) in some metrics, but still below expert levels.
- Baseline models like TF-IDF retrieval and end-to-end Seq2Seq perform poorly, while retrieval-augmented program generation models outperform them, demonstrating the importance of structured reasoning. FinBERT shows strong domain adaptation but still faces challenges in complex scenarios.
- Expert annotators achieve 87.49% accuracy, validating data quality. Non-experts perform significantly worse (~50%), confirming the task's difficulty and the necessity of domain expertise. Future work includes integrating external knowledge bases and improving multi-step reasoning accuracy.
Significance
This work addresses a critical gap in financial AI, providing a high-quality dataset for complex numerical reasoning over real-world financial documents. It advances the development of explainable AI systems capable of understanding heterogeneous financial data, facilitating automated financial analysis, risk assessment, and decision support, thus reducing reliance on manual labor and improving decision speed and accuracy.
Technical Contribution
The paper introduces a novel retrieval-generation architecture tailored for financial QA, combining expert-annotated reasoning programs with pre-trained language models. It emphasizes structured, multi-step reasoning with explicit operation sequences, enhancing interpretability. The dataset's detailed annotations and diverse question types set new standards for financial NLP research, enabling future innovations in domain-specific AI systems.
Novelty
FinQA is the first dataset focusing on multi-step numerical reasoning over financial reports, integrating expert-annotated reasoning programs with heterogeneous data. Unlike prior general-domain QA datasets, it emphasizes financial domain complexity, including heterogeneous data fusion and multi-operation reasoning, marking a significant step forward.
Limitations
- Models still struggle with multi-step reasoning and complex table structures, often generating incorrect or incomplete programs, limiting accuracy. The dataset, while large, cannot cover all financial scenarios, especially rare or novel report formats.
- Heavy reliance on domain-specific pretraining limits generalization to other financial contexts. Computational costs for training and inference remain high, hindering real-time applications.
- Current models lack robust external knowledge integration, which is crucial for understanding nuanced financial relationships and anomalies. Future work should focus on improving reasoning depth and knowledge incorporation.
Future Work
Future directions include integrating external financial knowledge graphs to improve reasoning, developing more scalable models for real-time analysis, and expanding the dataset to cover diverse financial report formats. Enhancing multi-modal understanding and explainability will also be key to broader adoption.
AI Executive Summary
Financial analysis is vital for informed decision-making in businesses and markets. However, the sheer volume and complexity of financial reports pose significant challenges for manual analysis, often requiring expert knowledge and extensive time. Traditional NLP approaches have made progress in extracting information or answering simple questions, but they fall short when faced with the intricate multi-step numerical reasoning needed for comprehensive financial understanding.
To address this gap, this paper introduces FinQA, a large-scale dataset comprising 8,281 question-answer pairs derived from real-world financial reports. These questions, crafted by financial experts, demand complex reasoning involving multiple operations such as addition, subtraction, multiplication, division, and table aggregation. Each question is paired with a detailed, annotated reasoning program, ensuring transparency and interpretability.
The core methodology employs a retrieval-augmented generation framework. First, a retriever based on BERT identifies relevant facts from the report. Then, a generator, leveraging models like RoBERTa, constructs a sequence of operations that, when executed, produce the correct answer. This approach mimics the step-by-step reasoning process used by financial analysts, translating it into a machine-readable format.
Experimental results demonstrate that the best model achieves 65.05% execution accuracy, significantly surpassing baseline methods but still far below human experts at 91.16%. These findings highlight the complexity of financial reasoning and the need for further model enhancements, such as better multi-step reasoning, external knowledge integration, and handling diverse report formats.
The significance of this work lies in providing a foundational resource for advancing AI in financial analysis. It opens avenues for developing explainable, accurate, and efficient financial AI systems capable of automating complex tasks, reducing manual effort, and supporting faster, more reliable decision-making. Despite current limitations, the dataset and framework set a new standard for research in domain-specific question answering and numerical reasoning, promising substantial industry impact in the coming years.
Deep Analysis
Background
财务文本分析经历了从简单信息抽取到复杂推理的逐步演变。早期研究主要集中在财务指标提取和比率分析（如盈利能力、偿债能力等），利用传统统计和机器学习方法。近年来，深度学习模型如BERT、RoBERTa在财务文本理解中表现出色，但多为分类或信息抽取任务，难以应对多步骤复杂推理。现有数据集如DROP和MathQA虽涉及数值计算，但缺乏财务专业知识的深度结合，无法满足财务场景的特殊需求。财务报告中的异构信息融合（表格与文本）、多操作推理等问题仍未得到充分解决。随着金融行业对自动化分析的需求增长，研发具有专业知识和可解释性的推理系统成为研究热点。
Core Problem
财务报告内容繁杂，信息分散在文本和多张表格中，人工分析费时费力，难以满足快速决策的需求。现有模型在理解财务表格关系、进行多步骤推理方面表现不足，缺乏透明的推理路径，限制了实际应用。核心问题在于如何构建能理解异构财务数据、进行深度推理且具备可解释性的自动系统，以减轻财务分析师的负担，提高分析效率和准确性。
Innovation
引入由财务专家标注的FinQA数据集，结合多操作符的推理程序，显著提升模型处理复杂财务问题的能力。提出检索-生成架构，利用预训练模型筛选支持事实，并逐步生成操作指令，模拟财务分析中的多步推理过程。创新点在于将财务知识结构化表达，确保推理过程透明，增强模型的可解释性。不同于传统单一任务或通用问答，强调财务专业知识的深度结合，推动财务AI向智能分析迈进。
Methodology
- �� 数据准备：从FinTabNet筛选财务报告，过滤掉结构复杂或内容不相关的页面。
- �� 标注问答：由财务专家设计问题，要求结合表格与文本信息，制定多步推理程序。
- �� 构建模型：采用检索模块（基于BERT）筛选支持事实，生成模块（基于RoBERTa）逐步构建推理程序。
- �� 程序表达：定义操作符（加减乘除、表格聚合）及参数，确保程序可执行。
- �� 训练流程：结合专家标注的推理路径，优化模型生成的准确性和可解释性。
- �� 评估指标：包括执行准确率（答案正确率）和程序等价性（结构匹配）评估。
- �� 实验设计：比较不同预训练模型、检索策略，进行消融分析验证方法有效性。
Experiments
采用FinQA数据集，划分训练、验证、测试集，模型在BERT、RoBERTa、FinBERT等预训练模型上训练。调优超参数（学习率、批次大小），评估模型在执行和程序准确率上的表现。与基线模型（TF-IDF检索、端到端Seq2Seq、NeRd）进行对比，验证结构化推理的重要性。还进行了专家与非专家标注的质量验证，确保数据可靠。
Results
最优模型（RoBERTa-large）在测试集达58.86%的程序准确率，显著优于非专家（50.68%），但仍低于专家（87.49%）。多步骤推理中，模型表现不稳定，验证了财务推理的复杂性。消融实验显示，结合检索与程序生成效果优于单一端到端模型，强调结构化路径的重要性。
Applications
该系统可应用于财务报告自动分析、财务风险监控、智能财务助手等场景。用户只需上传财务报告，系统即可自动生成推理路径，提供财务指标变化、风险预警等信息。未来结合知识图谱，将实现更智能的财务决策支持。
Limitations & Outlook
模型在多步骤复杂推理和异构数据融合方面仍存在不足，推理路径生成不够准确，泛化能力有限。财务知识依赖较强，面对新颖报告或异常情况时表现不佳。计算成本较高，未来需优化模型效率和扩展能力。
Plain Language Accessible to non-experts
想象你在厨房准备一道复杂的菜肴，里面有许多不同的材料和步骤。你需要理解每一步的操作，比如加盐、搅拌、烤箱温度，还要结合不同材料的表格信息。财务报告就像这份菜谱，里面既有文字描述，也有各种表格。要知道公司财务的变化，就像按照菜谱一步步操作，逐步算出最终的味道。这个系统就像一个聪明的厨师助手，能看懂复杂的菜谱，帮你自动算出结果，让你更快做出美味佳肴。
ELI14 Explained like you're 14
你知道做饭时，有时候需要看一份很复杂的食谱，里面写着每一步要放多少盐、油，还要用到一些表格里的数字。想象你有个超级帮手，他能看懂这些复杂的食谱，帮你算出最后的味道是不是正好。它不仅能看懂文字，还能理解表格里的数字，比如哪个月份的销售额最高，或者利润增长了多少。它会一步步帮你算，最后告诉你公司赚了多少钱，或者亏了多少。虽然它还不能完全替代厨师，但已经可以帮你省很多时间，让你更快做出好菜！
Abstract
The sheer volume of financial statements makes it difficult for humans to access and analyze a business's financials. Robust numerical reasoning likewise faces unique challenges in this domain. In this work, we focus on answering deep questions over financial data, aiming to automate the analysis of a large corpus of financial documents. In contrast to existing tasks on general domain, the finance domain includes complex numerical reasoning and understanding of heterogeneous representations. To facilitate analytical progress, we propose a new large-scale dataset, FinQA, with Question-Answering pairs over Financial reports, written by financial experts. We also annotate the gold reasoning programs to ensure full explainability. We further introduce baselines and conduct comprehensive experiments in our dataset. The results demonstrate that popular, large, pre-trained models fall far short of expert humans in acquiring finance knowledge and in complex multi-step numerical reasoning on that knowledge. Our dataset -- the first of its kind -- should therefore enable significant, new community research into complex application domains. The dataset and code are publicly available\url{https://github.com/czyssrs/FinQA}.