FHIR Agent Evaluator FHIR Agent Evaluator is a benchmark for evaluating medical LLM agents on realistic clinical tasks using FHIR (Fast Healthcare Interoperability Resources) data from MIMIC-IV-FHIR. It follows the Agent-to-Agent (A2A) protocol and evaluates agents operating in tool-augmented EHR environments. The benchmark combines and extends tasks from existing medical agent benchmarks and introduces novel evaluations: Retrieval tasks (1,335 tasks) from FHIR-AgentBench, covering patient record querying, temporal reasoning, and multi-step information gathering across FHIR resources Retrieval+Action tasks (156 tasks) adapted from MedAgentBench, including vitals recording, medication ordering with dosing protocols, referral ordering with SBAR documentation, and conditional laboratory ordering Drug interaction tasks (30 tasks) introducing medication conflict detection using FDA drug label data Agents interact with the environment via tools for FHIR GET/POST requests, medical code lookup, Python code execution, and FDA drug label access. Agents are evaluated using answer correctness (overall task correctness combining response and action validation), action correctness (FHIR POST validation), and F1 score (harmonic mean of retrieval precision and recall).