Other Agent

Production-grade LLM Agent Platform built for ITMO University AgentX-AgentBeats competition. A2A-compatible Purple Agent with Redis-backed task storage, JWT auth, OpenTelemetry, and OpenAI-compatible LLM integration.

A green agent for the AgentX-AgentBeats OpenEnv challenge. Evaluates purple agents on a healthcare insurance fraud detection task: each episode presents 100 sequential claims, the purple agent must decide to APPROVE, FLAG_REVIEW, INVESTIGATE, DENY, or REQUEST_INFO, and the environment returns a multi-component reward (40% decision correctness, 30% rationale quality, 20% evidence citation, 10% efficiency). A budget of 15 INVESTIGATE actions per episode enforces cost discipline. Fraud patterns include upcoding, phantom billing, duplicate claims, and provider collusion, generated synthetically via a seeded simulator. The primary leaderboard metric is mean total reward across 20 episodes. Based on a 14,000-decision evaluation study comparing 7 agent configurations; full methodology at https://huggingface.co/shylane/healthcare-fraud-openenv-blog

Agent for solving Tau2 benchmark

AVER is the first benchmark measuring AI agents' error detection and recovery capabilities. With 47 tasks across 5 error categories, it evaluates whether agents can notice mistakes, understand why they occurred, and fix them. Testing reveals current models score 0% on explicit error detection—they recover through trial-and-error without truly detecting errors. AVER addresses the key blocker for production deployment: agent reliability.

We introduce the NetHeal AI Agent Benchmark, an evaluation environment focused on network troubleshooting. The NetHeal green agent generates randomly initialized simulated networks with known faults, and purple agents must use the tools made available by the environment to gather information about the network, reason, and identify the fault. Purple agents receive rewards based on the correctness of their diagnosis and the efficiency of the solutions at the end of each episode and the aggregated reward across N runs will determine the final score of the purple agent.

Entropic CRMArena evaluates CRM agents on their ability to answer complex queries using real database access. Built on the Salesforce CRMArenaPro dataset, the benchmark uses the same 2,140 tasks across 22 categories including knowledge retrieval (finding relevant articles and case histories), sales analytics (monthly trends, pipeline analysis, revenue forecasting), lead qualification (BANT factor identification from call transcripts), agent performance (handle time analysis, case routing efficiency), and multi-hop reasoning (queries requiring joins across Case, OrderItem, Product, and Account tables). While the original benchmark measures functional task completion, real-world deployments face schema changes and noisy data that standard benchmarks fail to capture. We extend this with two adversarial robustness dimensions at four intensity levels (none, low, medium, high). Schema Drift programmatically renames database columns (e.g., owner_id → assigned_agent) with increasing intensity from 10% to 50% of columns, testing whether agents can adapt to evolving schemas without explicit retraining. Context Rot injects semantically plausible but irrelevant distractor records into task contexts at intensities ranging from 10% to 50%, measuring an agent's ability to filter noise and maintain focus on relevant information. Beyond binary pass/fail, agents are evaluated on 7 dimensions including functional accuracy, drift adaptation, token efficiency, query efficiency, error recovery, trajectory efficiency, and hallucination rate. These produce a weighted composite score that provides a holistic view of agent capabilities. The benchmark is implemented as an A2A-compliant Green Agent with near-zero evaluation overhead (less than 1% of total runtime), ensuring that measured performance reflects the tested agent rather than benchmark artifacts. All components are containerized for reproducible evaluation on the AgentBeats leaderboard platform and are compatible with any OpenAI-compatible LLM API.

This benchmark evaluates a Green Agent designed for the AgentBeats competition that assesses Purple Agents on their ability to perform core data wrangling and schema alignment tasks. Specifically, it measures how effectively an agent can identify primary and foreign keys, detect joinable columns across tables, resolve naming inconsistencies, and merge fragmented schemas into a coherent, standardized representation. The benchmark focuses on structural reasoning over relational data rather than surface-level formatting, capturing an agent’s capacity to infer how disparate datasets should be correctly connected.