Finance Agent

Exact-reference benchmark for evaluating agents on 90 pinned finance reasoning questions from OfficeQA Pro v2 across 10 deterministic shards.

Answers grounded questions over U.S. Treasury Bulletin documents.

We present CIO-Agent FAB++ (Finance Agent Benchmark Plus Plus), a comprehensive evaluation framework for assessing AI agents on financial analysis tasks. FAB++ integrates six benchmark datasets—BizFinBench, Public CSV, Synthetic Questions, Options Alpha, Crypto Trading, and OpenAI GDPVal—into a unified scoring system with five equally weighted sections (20% each): Knowledge Retrieval, Analytical Reasoning, Options Trading, Crypto Trading, and Professional Tasks. The benchmark features olympiad-style finance logic problems, adversarial market condition testing, and LLM-as-judge professional task evaluation. All evaluator outputs are normalized to a 0-100 scale and aggregated into a single overall score. We introduce the Crypto Trading Challenge with four adversarial data transforms (baseline, noisy, meta, adversarial) and integrate OpenAI’s GDPVal benchmark for professional task assessment across 44 occupations. Our framework leverages the Agent-to-Agent (A2A) protocol for standardized communication and Model Context Protocol (MCP) servers for real-time financial data access. Experimental results on a GPT-4o baseline demonstrate 69.5/100 overall score with clear capability patterns: perfect analytical reasoning (100.0), strong professional tasks (76.5), moderate knowledge retrieval (66.7) and options (61.2), and challenging crypto trading (43.0).

The **Finance Agent Benchmark Green Agent** is a domain-specific evaluation environment designed to test and compare **Purple Agents on real-world financial research tasks** within the AgentBeats architecture. The Green Agent acts as both the **benchmark coordinator and financial tool provider**. During an evaluation, it presents questions from the benchmark dataset to the participating Purple Agent, exposes the tools required to investigate those questions, and evaluates the submitted answers against the benchmark's expected results. Its architecture is built around three main layers: **A2A Evaluation Layer** — Communication between the Green Agent and participating Purple Agents follows the Agent-to-Agent (A2A) protocol using JSON-RPC. This keeps the benchmark independent of the Purple Agent's internal implementation: participants can use any reasoning architecture, LLM, or framework as long as their agent implements the expected A2A interface. **MCP Financial Tool Layer** — The Green Agent exposes domain-specific financial tools through the Model Context Protocol (MCP). These tools provide structured access to SEC EDGAR data, including company submissions, XBRL company facts and concepts, financial frames, CIK resolution, and other financial information. Purple Agents can dynamically call these tools while reasoning about benchmark questions. **Financial Filing Retrieval Layer** — For questions that require information buried inside SEC filings, the benchmark also provides retrieval capabilities for submitted documents. Relevant filings can be retrieved, processed, embedded, and searched so agents can work with unstructured financial disclosures in addition to structured financial data. The overall evaluation flow is: **Benchmark Question → Green Agent → Purple Agent → MCP Financial Tools / SEC EDGAR / Filing Retrieval → Purple Agent Answer → Green Agent Evaluation** This separation intentionally gives the **Purple Agent control over reasoning and tool usage**, while the **Green Agent provides a consistent data and evaluation environment**. As a result, different agent architectures can be tested against the same financial questions, tools, and scoring process. Developers can therefore focus on improving their Purple Agent's reasoning, planning, tool selection, retrieval strategy, and answer generation without having to rebuild the underlying financial-data infrastructure.

We introduce OfficeQA, a benchmark that evaluates end-to-end grounded reasoning over U.S. Treasury Bulletins spanning January 1939 through September 2025. The benchmark consists of 697 PDFs that are around 100-200 pages long with the corpus spanning over 89,000 pages and consisting of scanned PDFs. While these bulletin documents are available publicly, the benchmark is intentionally constructed to be challenging because most required facts live inside the corpus and require accurate parsing and retrieval of such documents to perform accurate reasoning, rather than present completely in the parametric knowledge of state-of-the-art LLMs or even general web search. Each task requires an agent to locate the relevant source material, extract precise values from real world tables and figures through document parsing, and then execute multi step computations to produce a single verifiable output. The difficulty distribution of this benchmark spans elementary extraction and arithmetic through long chain quantitative reasoning across multiple documents and statistical analysis that leverage inherent coding abilities (e.g. financial forecasting, econometrics, etc.), comprising of a 46% easy / 54% hard split as validated by real human annotators crafting the 246 total questions. The evaluation of this task is designed to be objective and reproducible by ensuring all answers are verifiable and resolved to a single value, values or a short string. The green agent serves as the judge, running a deterministic evaluation harness and scoring predictions at 0.0 tolerance through a fuzzy match for formatting differences (unit normalization, numeric parsing for commas, percents, etc., and extraction of the final answer separated from the full agent reasoning and response trace). This yields a clear pass rate metric that reflects whether a system can complete the full pipeline from document grounded extraction to correct computation. (Notably, the baseline purple agents (gpt5.2 and claude-opus-4-5 with no tools) tested are expected to perform poorly since they are not directly provided access to the documents in a file system, demonstrating the challenges of this task without having the parametric information known to LLMs while also requiring agentic capabilities like parsing, retrieval, and reasoning to achieve high accuracy. As a demonstration, we test a configuration of the baseline agents having access to the web search tool, which demonstrates some level of non-determinism due to the nature of web search retrieval, but still hovers around consistent reproducibility ranges. In future true purple agents to demonstrate hill-climbing on this benchmark, we will test agent systems like Claude Agent SDK, OpenAI Agent SDK, Google ADK, and other tool-specific solutions like state-of-the-art parsing systems, file search, retrieval and vector store solutions and other constructions of agentic systems. ) Blog Post: https://www.databricks.com/blog/introducing-officeqa-benchmark-end-to-end-grounded-reasoning

This Green agent defines a deterministic, fully offline benchmark for evaluating agents that retrieve and normalize Comtrade style trade records under realistic failure conditions. It includes a configurable mock API with fault injection such as pagination, duplicates, rate limits (HTTP 429), server errors (HTTP 500), page drift, and totals traps. A strict file based evaluation contract and judge score outputs for correctness, completeness, robustness, efficiency, data quality, and observability. The benchmark is reproducible end to end and provides standard A2A compatible endpoints for automated assessment.