Healthcare Agent

NurseSim-Triage evaluates an agent's ability to perform safety-critical clinical triage in Emergency Department scenarios. The agent receives patient presentations (chief complaint, vital signs, demographics, medical history) and must assign the correct Manchester Triage System category (1-5) while providing clinical reasoning. Tasks assess: Risk Stratification - Correctly identifying life-threatening conditions (Category 1: Cardiac arrest, Anaphylaxis, Sepsis) Demographic Context Integration - Weighing age and gender as risk modifiers (e.g., chest pain in 72M vs 20M) Safety-Critical Decision Making - Avoiding dangerous under-triage that could delay life-saving treatment Clinical Reasoning - Explaining triage decisions with medically sound rationale The benchmark includes 15 gold-standard scenarios spanning all 5 MTS categories, evaluated by GPT-5.2 judges for both accuracy and safety complia

The green agent evaluates AI models on 10 clinical reasoning tasks from Stanford MedAgentBench, testing capabilities in patient data queries, vital signs recording, laboratory analysis, medication management, and consultation ordering across standardized medical scenarios. The project also examines AI's ability to distinguish real pharmaceuticals from fabricated drug names, as explored in research titled "Drug or Pokemon?" This dual focus assesses both clinical workflow automation and AI safety in medical decision-making contexts.

FHIR Agent Evaluator FHIR Agent Evaluator is a benchmark for evaluating medical LLM agents on realistic clinical tasks using FHIR (Fast Healthcare Interoperability Resources) data from MIMIC-IV-FHIR. It follows the Agent-to-Agent (A2A) protocol and evaluates agents operating in tool-augmented EHR environments. The benchmark combines and extends tasks from existing medical agent benchmarks and introduces novel evaluations: Retrieval tasks (1,335 tasks) from FHIR-AgentBench, covering patient record querying, temporal reasoning, and multi-step information gathering across FHIR resources Retrieval+Action tasks (156 tasks) adapted from MedAgentBench, including vitals recording, medication ordering with dosing protocols, referral ordering with SBAR documentation, and conditional laboratory ordering Drug interaction tasks (30 tasks) introducing medication conflict detection using FDA drug label data Agents interact with the environment via tools for FHIR GET/POST requests, medical code lookup, Python code execution, and FDA drug label access. Agents are evaluated using answer correctness (overall task correctness combining response and action validation), action correctness (FHIR POST validation), and F1 score (harmonic mean of retrieval precision and recall).