Finance Agent

Original purple agent from https://github.com/arnavsinghvi11/officeqa_agentbeats ported to use amber.

We present an evaluator agent that leverages a custom-made, structured dataset of questions to assess large language models (LLMs) on financial reasoning and aggregation tasks over real-world exchange-traded fund (ETF) data. To construct this dataset and the associated agent, we developed a crawler that collects ETF documentation from major brokerages and asset managers, including Fidelity, Schwab, Vanguard, and BlackRock, and normalized the extracted information into per-ETF JSON files. The resulting corpus spans 641 ETFs, comprising 34 Fidelity ETFs, 471 BlackRock ETFs, 33 Schwab ETFs, and 103 Vanguard ETFs. Building on an initial set of question templates, we curated 300 question–answer pairs spanning four evaluation dimensions—fundamentals, performance and risk-adjusted returns, liquidity and trading, and cost and tax efficiency—with a focus on numeric, script-computable targets. These questions require filtering, counting, conditional reasoning, and aggregation over financial attributes such as valuation ratios, dividend and distribution metrics, returns and risk statistics, liquidity measures, and expense ratios, including summary statistics (e.g., mean/median/standard deviation) and quantile-based aggregation (e.g., top-quartile proportions) over provider-specific ETF universes. Each question is paired with a deterministic script that computes the ground-truth answer directly from the underlying JSON data, enabling reproducible and automated evaluation. We then use the evaluator agent to pose these questions to a target LLM and grade its responses via an agent-to-agent (A2A) protocol. Together, the dataset and evaluator agent support systematic assessment of LLM performance on financial data understanding.