Generative AI in organizations and society
AI financial claim verification still trails experts
FINDVER tests whether language models can locate and explain evidence in complex financial documents—and finds a gap between model outputs and expert judgment.
Financial claims often depend on figures buried in annual reports, regulatory filings and tables. A reader must find the right passages, connect them to a claim, calculate what is needed and explain the conclusion. FINDVER is a benchmark designed to test that whole process rather than only whether a final label is correct.
Three tasks inside a financial document
The benchmark separates claim verification into information extraction, numerical reasoning and knowledge-intensive reasoning. These subsets represent different work: locating relevant evidence, performing calculations across document elements, and combining evidence with financial-domain knowledge. Documents can contain long passages as well as tables, so success requires navigating mixed formats and giving a readable rationale (task design and comparison with earlier datasets, PDF pp. 1–3; proceedings pp. 14739–14741).
The authors curate claims and evidence from financial documents, then ask annotators to provide entailment labels and step-by-step explanations. In this context, “entailment” means the evidence supports a claim; a contradiction or lack of sufficient evidence calls for a different decision. A separate validation process checks whether cited evidence is relevant and complete, whether the label follows from it, and whether the explanation applies the right reasoning. The paper reports that 347 of 2,100 initial examples were revised during validation. Two evaluators also rated a 100-example sample; Table 2 summarizes those quality checks (PDF pp. 3–5; proceedings pp. 14741–14743).
A large context window is not expert judgment
The researchers assess open and closed language models in long-context and retrieval-augmented generation settings. Retrieval-augmented generation, or RAG, first searches a collection for passages and then gives those passages to a model to answer. The experiments show that even the strongest system they tested, Claude 3.5 Sonnet, remains substantially behind human experts on the benchmark (abstract and model evaluation, PDF pp. 1 and 6–9; proceedings pp. 14739, 14744–14747).
The benchmark’s explanations make the gap easier to diagnose. A model can retrieve a plausible excerpt but miss a second number needed for a calculation, or produce a conclusion without a traceable chain of evidence. That distinction is practically important: a confident label is not the same as an auditable explanation.
Limits and possible uses
FINDVER concentrates on financial-document scenarios and its annotation choices, so results do not represent every kind of financial work or every system configuration. Its authors discuss limits to coverage and evaluation in the paper’s limitations section (PDF pp. 13–14; proceedings pp. 14751–14752). The benchmark does not establish that any tested model is ready to make investment or compliance decisions without review.
For teams that use AI to summarize reports, FINDVER suggests evaluating evidence selection and reasoning separately from answer fluency. Its task structure may help researchers locate failure points before a system is used in higher-stakes workflows.
Bibliography & sources
- Yilun Zhao, Yitao Long, Yuru Jiang, Chengye Wang, Weiyuan Chen, Hongjun Liu, Yiming Zhang, Xiangru Tang, Chen Zhao, and Arman Cohan. “FinDVer: Explainable Claim Verification over Long and Hybrid-content Financial Documents.” In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 14739–14752 (2024). https://doi.org/10.18653/v1/2024.emnlp-main.818.