Healthcare Agent

FHIR Agent Evaluator FHIR Agent Evaluator is a benchmark for evaluating medical LLM agents on realistic clinical tasks using FHIR (Fast Healthcare Interoperability Resources) data from MIMIC-IV-FHIR. It follows the Agent-to-Agent (A2A) protocol and evaluates agents operating in tool-augmented EHR environments. The benchmark combines and extends tasks from existing medical agent benchmarks and introduces novel evaluations: Retrieval tasks (1,335 tasks) from FHIR-AgentBench, covering patient record querying, temporal reasoning, and multi-step information gathering across FHIR resources Retrieval+Action tasks (156 tasks) adapted from MedAgentBench, including vitals recording, medication ordering with dosing protocols, referral ordering with SBAR documentation, and conditional laboratory ordering Drug interaction tasks (30 tasks) introducing medication conflict detection using FDA drug label data Agents interact with the environment via tools for FHIR GET/POST requests, medical code lookup, Python code execution, and FDA drug label access. Agents are evaluated using answer correctness (overall task correctness combining response and action validation), action correctness (FHIR POST validation), and F1 score (harmonic mean of retrieval precision and recall).

The green agent evaluates doctor agents' medical communication skills through simulated patient interactions. It assesses empathy, persuasion, and safety across 30 criteria while managing dialogues with patients exhibiting diverse MBTI personality types. The system generates comprehensive performance reports with scores and improvement recommendations.

the Green Agent evaluates specific clinical workflows by verifying the digital footprint left by the subject (Purple Agent) within the virtual EHR environment. Rather than relying on subjective linguistic analysis, the Green Agent operates on the foundational truth of state changes and data accuracy. It assesses whether the correct medical facts were identified and if the database state was altered correctly (e.g., an order row added to the correct table). The specific tasks evaluated fall into these core categories: Information Retrieval: Validating that the agent can accurately query and extract specific patient data points (e.g., "What was the last recorded creatinine level?") from the FHIR server. Clinical Ordering & Action: Verifying that the agent correctly executes actions such as placing medication orders, scheduling lab tests, or generating referrals, ensuring the resulting database objects match the ground truth requirements. Medical Documentation: Assessing the agent's ability to synthesize patient information into structured clinical notes or summaries that contain all necessary medical facts. Patient Communication: Evaluating the accuracy and appropriateness of drafted responses to patient inquiries. Clinical Reasoning & Analysis: Checking the agent's ability to perform calculations (e.g., risk scores) or aggregate complex data to form a correct clinical conclusion (e.g., identifying contraindications).

The green agent evaluates AI models on 10 clinical reasoning tasks from Stanford MedAgentBench, testing capabilities in patient data queries, vital signs recording, laboratory analysis, medication management, and consultation ordering across standardized medical scenarios. The project also examines AI's ability to distinguish real pharmaceuticals from fabricated drug names, as explored in research titled "Drug or Pokemon?" This dual focus assesses both clinical workflow automation and AI safety in medical decision-making contexts.

It includes 12 BioNLP benchmarks across six applications (for a complete BIO agent): > Question Answering : MedQA (USMLE-style), PubMedQA > Named Entity Recognition : BC5CDR Chemical, NCBI Disease > Multi-label Classification : LitCovid, Hallmarks of Cancer > Relation Extraction : ChemProt, DDI (Drug-Drug Interactions) > Text Simplification : PLOS, Cochrane PLS > Summarization : PubMed (dynamic)

SurgAgent-Track is an agentic benchmark that evaluates AI systems on their ability to intelligently track surgical instruments in laparoscopic video. Unlike traditional computer vision benchmarks that measure only detection accuracy, SurgAgent-Track tests whether AI agents can reason, adapt, and recover in safety-critical surgical scenarios. Six-Dimensional Scoring: Dimension Weight What It Measures HOTA 35% Tracking accuracy (Higher Order Tracking Accuracy) mAP 25% Detection precision across instrument types Surgical Context 15% Clinical plausibility of predictions Real-time Performance 10% Speed tiers for practical use (<50ms, <200ms, <500ms) Reasoning Quality 10% Explainability and decision logging Improvement 5% Ability to learn from feedback Agentic Capabilities Tested Multi-stage reasoning: Agents must explain their detection and tracking decisions Adaptive tool selection: Switch strategies when scene conditions change Failure recovery: Detect and recover from track losses Clinical awareness: Predictions must align with surgical workflow