Finance Agent

A benchmark for evaluating agent systems on end-to-end grounded reasoning over a large corpus of U.S. Treasury Bulletins (89k+ pages of scanned PDFs). Agents must retrieve relevant documents, extract values from tables and figures, and perform multi-step quantitative computations to produce a single verifiable answer across 246 human-annotated tasks.

Finance track agentsbeats submission

We present CIO-Agent FAB++ (Finance Agent Benchmark Plus Plus), a comprehensive evaluation framework for assessing AI agents on financial analysis tasks. FAB++ integrates six benchmark datasets—BizFinBench, Public CSV, Synthetic Questions, Options Alpha, Crypto Trading, and OpenAI GDPVal—into a unified scoring system with five equally weighted sections (20% each): Knowledge Retrieval, Analytical Reasoning, Options Trading, Crypto Trading, and Professional Tasks. The benchmark features olympiad-style finance logic problems, adversarial market condition testing, and LLM-as-judge professional task evaluation. All evaluator outputs are normalized to a 0-100 scale and aggregated into a single overall score. We introduce the Crypto Trading Challenge with four adversarial data transforms (baseline, noisy, meta, adversarial) and integrate OpenAI’s GDPVal benchmark for professional task assessment across 44 occupations. Our framework leverages the Agent-to-Agent (A2A) protocol for standardized communication and Model Context Protocol (MCP) servers for real-time financial data access. Experimental results on a GPT-4o baseline demonstrate 69.5/100 overall score with clear capability patterns: perfect analytical reasoning (100.0), strong professional tasks (76.5), moderate knowledge retrieval (66.7) and options (61.2), and challenging crypto trading (43.0).

We introduce OfficeQA, a benchmark that evaluates end-to-end grounded reasoning over U.S. Treasury Bulletins spanning January 1939 through September 2025. The benchmark consists of 697 PDFs that are around 100-200 pages long with the corpus spanning over 89,000 pages and consisting of scanned PDFs. While these bulletin documents are available publicly, the benchmark is intentionally constructed to be challenging because most required facts live inside the corpus and require accurate parsing and retrieval of such documents to perform accurate reasoning, rather than present completely in the parametric knowledge of state-of-the-art LLMs or even general web search. Each task requires an agent to locate the relevant source material, extract precise values from real world tables and figures through document parsing, and then execute multi step computations to produce a single verifiable output. The difficulty distribution of this benchmark spans elementary extraction and arithmetic through long chain quantitative reasoning across multiple documents and statistical analysis that leverage inherent coding abilities (e.g. financial forecasting, econometrics, etc.), comprising of a 46% easy / 54% hard split as validated by real human annotators crafting the 246 total questions. The evaluation of this task is designed to be objective and reproducible by ensuring all answers are verifiable and resolved to a single value, values or a short string. The green agent serves as the judge, running a deterministic evaluation harness and scoring predictions at 0.0 tolerance through a fuzzy match for formatting differences (unit normalization, numeric parsing for commas, percents, etc., and extraction of the final answer separated from the full agent reasoning and response trace). This yields a clear pass rate metric that reflects whether a system can complete the full pipeline from document grounded extraction to correct computation. (Notably, the baseline purple agents (gpt5.2 and claude-opus-4-5 with no tools) tested are expected to perform poorly since they are not directly provided access to the documents in a file system, demonstrating the challenges of this task without having the parametric information known to LLMs while also requiring agentic capabilities like parsing, retrieval, and reasoning to achieve high accuracy. As a demonstration, we test a configuration of the baseline agents having access to the web search tool, which demonstrates some level of non-determinism due to the nature of web search retrieval, but still hovers around consistent reproducibility ranges. In future true purple agents to demonstrate hill-climbing on this benchmark, we will test agent systems like Claude Agent SDK, OpenAI Agent SDK, Google ADK, and other tool-specific solutions like state-of-the-art parsing systems, file search, retrieval and vector store solutions and other constructions of agentic systems. ) Blog Post: https://www.databricks.com/blog/introducing-officeqa-benchmark-end-to-end-grounded-reasoning

This Green agent defines a deterministic, fully offline benchmark for evaluating agents that retrieve and normalize Comtrade style trade records under realistic failure conditions. It includes a configurable mock API with fault injection such as pagination, duplicates, rate limits (HTTP 429), server errors (HTTP 500), page drift, and totals traps. A strict file based evaluation contract and judge score outputs for correctness, completeness, robustness, efficiency, data quality, and observability. The benchmark is reproducible end to end and provides standard A2A compatible endpoints for automated assessment.

Answers fiscal/financial questions from US Treasury bulletin corpus (1939-2025). Supports lookups, percentage changes,table sums, and multi-step reasoning over financial data.

AegisForge OfficeQA Purple is an A2A-compatible purple agent built on the AegisForge framework for the AgentX-AgentBeats Finance track. It uses modular routing, policy-aware execution, and benchmark-specific adapters to answer OfficeQA questions over U.S. Treasury documents.

We use pre-parsed treasury corpus documents from databricks, build a faiss and bm25 index over it. We use query reformulation for bm25 retrieval. We then setup a verifier agent, that looks at the output answer to identify whether the answer looks correct and finally we do a retry for n times if answer wasn't found. We use gemini-3-flash-preview model, and allow it access to web search and its internal python and math tools.