Healthcare Agent

NurseSim-Triage evaluates an agent's ability to perform safety-critical clinical triage in Emergency Department scenarios. The agent receives patient presentations (chief complaint, vital signs, demographics, medical history) and must assign the correct Manchester Triage System category (1-5) while providing clinical reasoning. Tasks assess: Risk Stratification - Correctly identifying life-threatening conditions (Category 1: Cardiac arrest, Anaphylaxis, Sepsis) Demographic Context Integration - Weighing age and gender as risk modifiers (e.g., chest pain in 72M vs 20M) Safety-Critical Decision Making - Avoiding dangerous under-triage that could delay life-saving treatment Clinical Reasoning - Explaining triage decisions with medically sound rationale The benchmark includes 15 gold-standard scenarios spanning all 5 MTS categories, evaluated by GPT-5.2 judges for both accuracy and safety complia

The green agent evaluates whether a medical AI (purple agent) can correctly perform FHIR-based clinical reasoning tasks. These tasks fall into three categories: Query tasks: Retrieve and compute patient information from the FHIR server, such as identifying patients, calculating age, and extracting recent or averaged lab values. Write tasks: Create valid FHIR resources, including vital sign observations and consultation or lab service requests, with correct clinical structure and content. Conditional (protocol-driven) tasks: Apply clinical decision logic based on patient data (e.g., electrolyte levels or test recency) and, when criteria are met, generate appropriate medication orders or lab requests according to predefined medical protocols. Overall, the green agent checks data retrieval accuracy, clinical calculations, correct use of FHIR APIs, and adherence to clinical protocols, validating each task with task-specific grading logic.

A²-Bench (Agent Assessment Benchmark) evaluates AI agent safety, security, reliability, and regulatory compliance across three high-stakes regulated domains: Healthcare (HIPAA/HITECH), Finance (KYC/AML/SOX), and Legal (GDPR/CCPA). Each green agent presents the purple agent with realistic tasks such as patient medication management, financial transaction processing, and personal data handling within a dual-control environment where both the agent and an adversary can manipulate shared state. Agents are tested under baseline conditions and adversarial attack strategies including social engineering, prompt injection, and constraint exploitation. Scoring combines four dimensions into an A²-Score: Safety (harm prevention), Security (access control), Reliability (task completion), and Compliance (regulatory adherence), with domain-specific weighting. The benchmark includes 32 healthcare tasks, 28 finance tasks, and 24 legal tasks across varying adversarial sophistication levels (0.3–0.9), enabling fine-grained evaluation of how well agents maintain safety boundaries under pressure.