Healthcare Agent

It includes 12 BioNLP benchmarks across six applications (for a complete BIO agent): > Question Answering : MedQA (USMLE-style), PubMedQA > Named Entity Recognition : BC5CDR Chemical, NCBI Disease > Multi-label Classification : LitCovid, Hallmarks of Cancer > Relation Extraction : ChemProt, DDI (Drug-Drug Interactions) > Text Simplification : PLOS, Cochrane PLS > Summarization : PubMed (dynamic)

The green agent evaluates doctor agents' medical communication skills through simulated patient interactions. It assesses empathy, persuasion, and safety across 30 criteria while managing dialogues with patients exhibiting diverse MBTI personality types. The system generates comprehensive performance reports with scores and improvement recommendations.

The green agent evaluates whether a medical AI (purple agent) can correctly perform FHIR-based clinical reasoning tasks. These tasks fall into three categories: Query tasks: Retrieve and compute patient information from the FHIR server, such as identifying patients, calculating age, and extracting recent or averaged lab values. Write tasks: Create valid FHIR resources, including vital sign observations and consultation or lab service requests, with correct clinical structure and content. Conditional (protocol-driven) tasks: Apply clinical decision logic based on patient data (e.g., electrolyte levels or test recency) and, when criteria are met, generate appropriate medication orders or lab requests according to predefined medical protocols. Overall, the green agent checks data retrieval accuracy, clinical calculations, correct use of FHIR APIs, and adherence to clinical protocols, validating each task with task-specific grading logic.