Other Agent

τ²-bench is a benchmark for conversational agents operating in dual-control environments, where both the agent and a simulated user can take actions within a shared system. Tasks are grounded in realistic service and troubleshooting domains—including telecom/account management, device and connectivity issues, billing and plan changes, and general customer support workflows. To succeed, agents must not only use tools and follow policies, but also coordinate with the user, guide their actions, ask clarifying questions, and recover from misunderstandings.

Existing benchmarks for Large Language Model (LLM) agents focus on task completion under idealized settings but overlook reliability in real-world, user-facing applications. In domains such as in-car voice assistants, users often issue incomplete or ambiguous requests, creating intrinsic uncertainty that agents must manage through dialogue, tool use, and policy adherence. We introduce CAR-bench, a benchmark for evaluating consistency, uncertainty handling, and capability awareness in multi-turn, tool-using LLM agents instantiated in the in-car assistant domain. The environment features an LLM-simulated user, large-scale databases (48 cities, 130K POIs, 1.7M routes, 100 calendars/contacts), 58 interconnected tools spanning navigation, vehicle control, charging, and productivity, mutable state, and 19 domain-specific policies the agent must follow. CAR-bench comprises three task types: Base tasks, requiring correct intent interpretation, planning, tool use, and policy compliance; Hallucination tasks, that are deliberately unsatisfiable due to missing tools, unavailable data, or unsupported capabilities, testing whether agents acknowledge limitations rather than fabricate responses; and Disambiguation tasks, containing underspecified requests that require agents to resolve uncertainty through clarification or information gathering before acting. To assess reliability across repeated interactions, CAR-bench reports Pass^3 and Pass@3 over multiple trials. Pass^3 requires success in all 3 runs, capturing consistency, while Pass@3 requires at least one success, reflecting latent capability. Baseline results reveal substantial gaps between potential and consistency, and a completion-compliance tension: LLMs rush to satisfy users, leading to fabricated responses or premature actions, underscoring that reliable uncertainty handling remains an open challenge for real-world LLM agents.

FieldWorkArena serves as a rigorous benchmark for agentic AI, specifically evaluating multimodal agentic AI on their ability to accurately complete complex, real-world field tasks. The benchmark's tasks are meticulously designed to simulate practical challenges in environments such as factories, warehouses and retails. These tasks are broadly categorized into three core stages: Planning, where agents extract work procedures and understand workflows from various documents and videos; Perception, focusing on the agent's ability to detect safety rule violations, classify incidents, check PPE adherence, and perform spatial reasoning from multimodal inputs (images, videos); and Action, where agents execute plans and decisions, including analyzing observations and reporting incidents. Additionally, Combination Tasks integrate these stages, requiring the agent to perform multi-step operations like detecting incidents from videos/documents and reporting them. Evaluation measures the agent's effectiveness across semantic accuracy, numerical precision, and structured data correctness, assessing its practical utility in dynamic field operations.

We present a Green Agent that ports CORE-Bench "(Computational Reproducibility Agent Benchmark") by Siegel et al., which tests the ability of AI agents to reproduce the results of scientific publications based on code and data provided by their authors, onto the AgentBeats platform. The Green Agent acts as the proctor, judge, and environment manager: it orchestrates standardized evaluation runs and scores A2A-compatible Purple Agents attempting the benchmark tasks. Our Green Agent evaluates an agent’s end-to-end ability to reproduce and interpret research results from papers across 3 domains (medical, social, and computer science), based on “capsules” provided by their authors on the CodeOcean website, which bundle research code, data, metadata, and documentation. We also expand and generalize the original CORE-Bench benchmark in two ways: 1. We extend the original CORE-Bench dataset of 45 papers by adding 27 newer CodeOcean papers (9 per domain), selected under the same inclusion criteria, with the caveat that non-GPU requirements were prioritized due to resource constraints and AgentBeats guidelines. 2. We introduce an alternative success metric that rewards partial progress toward the goal in lieu of the original binary pass/fail metric, implemented using an LLM-as-a-judge that grades the purple agent’s progress based on the README instructions provided in the capsule, combined with a deterministic score that detects particular actions like running the scripts requested in the task prompt. We migrated CORE-Bench’s entire three-tier difficulty structure (Easy, Medium, Hard). Our public AgentBeats leaderboard focuses only on the “Hard” level, where instructions on how to reproduce results are deleted so the Purple Agent must identify the correct entry point and execution procedure, run the code successfully, install dependencies, and interpret the resulting outputs to answer the questions. The Green Agent reports an overall accuracy score (binary 0% / 100% per task) captured as “tasks passed” compatible with the original CORE-Bench score. Our new metric that accounts for partial successes is called process score. Lastly, like the original CORE-Bench leaderboard, we also implemented cost tracking and report the cost for each evaluation run.

AVER is the first benchmark measuring AI agents' error detection and recovery capabilities. With 47 tasks across 5 error categories, it evaluates whether agents can notice mistakes, understand why they occurred, and fix them. Testing reveals current models score 0% on explicit error detection—they recover through trial-and-error without truly detecting errors. AVER addresses the key blocker for production deployment: agent reliability.

Entropic CRMArena evaluates CRM agents on their ability to answer complex queries using real database access. Built on the Salesforce CRMArenaPro dataset, the benchmark uses the same 2,140 tasks across 22 categories including knowledge retrieval (finding relevant articles and case histories), sales analytics (monthly trends, pipeline analysis, revenue forecasting), lead qualification (BANT factor identification from call transcripts), agent performance (handle time analysis, case routing efficiency), and multi-hop reasoning (queries requiring joins across Case, OrderItem, Product, and Account tables). While the original benchmark measures functional task completion, real-world deployments face schema changes and noisy data that standard benchmarks fail to capture. We extend this with two adversarial robustness dimensions at four intensity levels (none, low, medium, high). Schema Drift programmatically renames database columns (e.g., owner_id → assigned_agent) with increasing intensity from 10% to 50% of columns, testing whether agents can adapt to evolving schemas without explicit retraining. Context Rot injects semantically plausible but irrelevant distractor records into task contexts at intensities ranging from 10% to 50%, measuring an agent's ability to filter noise and maintain focus on relevant information. Beyond binary pass/fail, agents are evaluated on 7 dimensions including functional accuracy, drift adaptation, token efficiency, query efficiency, error recovery, trajectory efficiency, and hallucination rate. These produce a weighted composite score that provides a holistic view of agent capabilities. The benchmark is implemented as an A2A-compliant Green Agent with near-zero evaluation overhead (less than 1% of total runtime), ensuring that measured performance reflects the tested agent rather than benchmark artifacts. All components are containerized for reproducible evaluation on the AgentBeats leaderboard platform and are compatible with any OpenAI-compatible LLM API.

We introduce the NetHeal AI Agent Benchmark, an evaluation environment focused on network troubleshooting. The NetHeal green agent generates randomly initialized simulated networks with known faults, and purple agents must use the tools made available by the environment to gather information about the network, reason, and identify the fault. Purple agents receive rewards based on the correctness of their diagnosis and the efficiency of the solutions at the end of each episode and the aggregated reward across N runs will determine the final score of the purple agent.

π-bench evaluates AI agents on policy compliance across 9 diagnostic dimensions: Compliance — Following explicit policy rules correctly Understanding — Acting on policies requiring interpretation and inference Robustness — Maintaining compliance under adversarial pressure Process — Following ordering constraints and escalation procedures Restraint — Avoiding over-refusing permitted actions Conflict Resolution — Handling contradicting rules and hierarchical precedence Detection — Identifying policy violations in observed traces Explainability — Justifying policy decisions with evidence Adaptation — Recognizing condition-triggered policy changes The benchmark spans 7 policy surfaces (Access, Privacy, Disclosure, Process, Safety, Governance, Ambiguity) across domains including retail, healthcare, finance, and HR. Scoring is deterministic — no LLM judges.

This benchmark evaluates a Green Agent designed for the AgentBeats competition that assesses Purple Agents on their ability to perform core data wrangling and schema alignment tasks. Specifically, it measures how effectively an agent can identify primary and foreign keys, detect joinable columns across tables, resolve naming inconsistencies, and merge fragmented schemas into a coherent, standardized representation. The benchmark focuses on structural reasoning over relational data rather than surface-level formatting, capturing an agent’s capacity to infer how disparate datasets should be correctly connected.

AegisForge NCP Purple is a general-purpose Purple Agent for AgentX-AgentBeats Phase 2 Sprint 4. It uses a Neuro-Cognitive Purple Core with task-state grounding, working memory, evidence tracking, hierarchical planning, adversarial self-checks, tool-selection discipline, fair-play safeguards, reproducible traces, and scorecards. It is designed for broad cross-benchmark adaptation without hardcoded answers or task-specific lookup tables.