FinRank: An Evidence-Grounded Benchmark for Financial Question Answering and Retrieval over SEC Filings

The evolution of financial NLP has transitioned from simple keyword matching to sophisticated models capable of numerical reasoning and multi-modal understanding. Early datasets like FinQA and ConvFinQA focused on reasoning over structured financial data, primarily evaluating answer correctness. With the advent of large language models, retrieval-augmented generation (RAG) frameworks have gained popularity, enabling models to access extensive document corpora for answer synthesis. However, these approaches often lack rigorous evaluation of evidence provenance, especially in templated, repetitive SEC filings where distinguishing true support from confusable passages is critical. Existing benchmarks such as FinanceBench and FinDER address retrieval or correctness separately but do not incorporate curated hard negatives explicitly designed to challenge models’ discrimination ability. This gap hampers progress toward trustworthy, source-verified financial AI systems, which are essential for regulatory compliance, auditing, and high-stakes decision-making.

The core challenge in financial QA lies in the high similarity of disclosures across firms, periods, and document sections. Models often retrieve passages that appear relevant but lack true support, risking misinformation. Existing metrics overlook the importance of provenance, leading to overestimated performance. The difficulty is compounded by the dense, lengthy, and templated nature of SEC filings, which contain legal language, footnotes, and tables spread across multiple non-adjacent sections. The key problem is developing a benchmark and evaluation methodology that explicitly tests a model’s ability to identify and discriminate true evidence from confusable, distractor passages, thereby ensuring answer trustworthiness and interpretability.

FinRank’s main innovation is the manual curation of hard negatives—passages that are highly confusable with true supporting evidence—per question, enabling direct assessment of a model’s discrimination ability. It introduces a multi-dimensional annotation scheme covering difficulty, reasoning type, evidence scope, and sub-question decomposition, facilitating detailed performance analysis. The dataset combines sparse (BM25), dense (bi-encoder), and cross-encoder reranking architectures, with a focus on provenance and confusability. Unlike prior datasets, FinRank explicitly evaluates the ability to suppress distractors, providing a more realistic and industry-relevant benchmark for trustworthy financial NLP. This approach aligns with the need for models that can reliably trace answers back to their source disclosures.

• �� Data collection: Human annotators manually extracted question-answer pairs from SEC filings, identifying support passages and selecting confusable distractors.
• �� Annotation schema: Defined categories for difficulty, reasoning type (qualitative/quantitative), evidence scope (single/multi-passage), and sub-question decomposition.
• �� Hard negative selection: Human experts chose distractors from comparable filings, often from competitors or different periods, to simulate real-world confusions.
• �� Retrieval pipeline: Combined BM25 for initial candidate retrieval, bi-encoder models for dense embedding-based retrieval, and cross-encoder rerankers for final ranking.
• �� Evaluation metrics: Recall@10, MRR, nDCG@10 for retrieval; pairwise accuracy for discrimination.
• �� Experiments: Tested multiple models, analyzed performance degradation with hard negatives, and performed ablation studies to understand contributions of each component.

The experiments utilized the curated dataset of 1185 question-answer pairs, covering diverse sectors and difficulty levels. Baseline models included BM25, Sentence-BERT variants, and DeBERTa-based cross-encoders. Evaluation involved measuring Recall@10, MRR, and pairwise accuracy, comparing performance with random versus curated hard negatives. Results showed significant performance drops when hard negatives were introduced, confirming their challenge. Additional experiments varied model size (up to 7B parameters), reasoning types, and evidence scope, providing insights into model robustness and limitations. The setup ensured reproducibility, with detailed hyperparameters and evaluation protocols documented.

Models achieved only 44.8% Recall@10 with the best large encoder, highlighting the difficulty. Hard negatives caused a 13-20% decrease in pairwise accuracy, demonstrating their confusability. BM25 outperformed some small models, but overall, the results underscored the need for improved discrimination techniques. Ablation studies revealed that integrating sub-question decomposition and multi-passage reasoning significantly improved performance, but challenges remain in complex, multi-evidence scenarios. These findings validate the importance of curated negatives for realistic evaluation.

FinRank can be used to develop and benchmark financial QA systems that require high provenance fidelity, such as automated auditing, compliance monitoring, and financial reporting tools. Its emphasis on evidence discrimination makes it suitable for regulatory environments demanding source traceability. Practitioners can leverage the dataset to train models that better differentiate true disclosures from confusable information, ultimately enhancing trustworthiness and operational safety in financial AI applications.

The dataset’s limited scope—covering only 22 firms—may restrict generalization across broader sectors. Hard negatives are predominantly industry-internal, possibly missing other confusable types like legal or regulatory language. The evaluation focuses on retrieval metrics, not on the correctness or logical consistency of generated answers. Computational costs for large models remain high, and the manual annotation process limits scalability. Future work should address these issues by expanding data diversity, automating parts of annotation, and integrating answer quality assessments.