Healthcare Agent

This project is a Python benchmark framework for assessing healthcare agents that generate daily care plans based on Long-Term Care Insurance (LTCI) assessments. The system evaluates the quality of care plans across the following dimensions: Mandatory Task Coverage (50%), Safety Constraints (20%), Duration Reasonableness (30%) and Qualification Matching.

SurgAgent-Track is an agentic benchmark that evaluates AI systems on their ability to intelligently track surgical instruments in laparoscopic video. Unlike traditional computer vision benchmarks that measure only detection accuracy, SurgAgent-Track tests whether AI agents can reason, adapt, and recover in safety-critical surgical scenarios. Six-Dimensional Scoring: Dimension Weight What It Measures HOTA 35% Tracking accuracy (Higher Order Tracking Accuracy) mAP 25% Detection precision across instrument types Surgical Context 15% Clinical plausibility of predictions Real-time Performance 10% Speed tiers for practical use (<50ms, <200ms, <500ms) Reasoning Quality 10% Explainability and decision logging Improvement 5% Ability to learn from feedback Agentic Capabilities Tested Multi-stage reasoning: Agents must explain their detection and tracking decisions Adaptive tool selection: Switch strategies when scene conditions change Failure recovery: Detect and recover from track losses Clinical awareness: Predictions must align with surgical workflow

This Green Agent evaluates participant agents on their ability to perform safe and accurate clinical triage across a benchmark of 100 synthesized medical scenarios. The scenarios include a mix of Emergency (ex. stroke symptoms, severe allergic reactions) and Non-Emergency (ex. mild cold, minor sprains) cases. The evaluation process focuses on two primary metrics: Safety (Critical): Determines if the participant correctly identifies emergencies by checking for mandatory keywords (ex. "Call 911", "ER") and avoids dangerous advice in non-emergent cases (ex."ignore it", specific unverified dosage recommendations). Unsafe responses are immediately penalized with a score of 0. Helpfulness: Assesses whether the participant provides actionable follow-up advice for safely managed conditions (ex. "monitor symptoms", "contact primary care physician"). Each scenario is scored on a binary Pass/Fail basis derived from these metrics. The final leaderboard score reflects the agent's Accumulated Helpfulness Accuracy strictly gated by Clinical Safety. The evaluation also measures response latency to ensure timely triage guidance.