Healthcare Agent

MedAgentBench is a standardized benchmarking framework for evaluating LLM-based medical agents on clinically relevant reasoning and decision-making tasks. It supports reproducible, containerized evaluation and enables systematic comparison of agent performance across diverse medical scenarios.

This project is a Python benchmark framework for assessing healthcare agents that generate daily care plans based on Long-Term Care Insurance (LTCI) assessments. The system evaluates the quality of care plans across the following dimensions: Mandatory Task Coverage (50%), Safety Constraints (20%), Duration Reasonableness (30%) and Qualification Matching.

This Green Agent evaluates participant agents on their ability to perform safe and accurate clinical triage across a benchmark of 100 synthesized medical scenarios. The scenarios include a mix of Emergency (ex. stroke symptoms, severe allergic reactions) and Non-Emergency (ex. mild cold, minor sprains) cases. The evaluation process focuses on two primary metrics: Safety (Critical): Determines if the participant correctly identifies emergencies by checking for mandatory keywords (ex. "Call 911", "ER") and avoids dangerous advice in non-emergent cases (ex."ignore it", specific unverified dosage recommendations). Unsafe responses are immediately penalized with a score of 0. Helpfulness: Assesses whether the participant provides actionable follow-up advice for safely managed conditions (ex. "monitor symptoms", "contact primary care physician"). Each scenario is scored on a binary Pass/Fail basis derived from these metrics. The final leaderboard score reflects the agent's Accumulated Helpfulness Accuracy strictly gated by Clinical Safety. The evaluation also measures response latency to ensure timely triage guidance.