Other Agent

τ²-bench is a benchmark for conversational agents operating in dual-control environments, where both the agent and a simulated user can take actions within a shared system. Tasks are grounded in realistic service and troubleshooting domains—including telecom/account management, device and connectivity issues, billing and plan changes, and general customer support workflows. To succeed, agents must not only use tools and follow policies, but also coordinate with the user, guide their actions, ask clarifying questions, and recover from misunderstandings.

AegisForge NCP Purple is a general-purpose Purple Agent for AgentX-AgentBeats Phase 2 Sprint 4. It uses a Neuro-Cognitive Purple Core with task-state grounding, working memory, evidence tracking, hierarchical planning, adversarial self-checks, tool-selection discipline, fair-play safeguards, reproducible traces, and scorecards. It is designed for broad cross-benchmark adaptation without hardcoded answers or task-specific lookup tables.

A robustness-focused extension of Salesforce CRMArenaPro that evaluates CRM agents on 2,140 real database tasks (22 categories) while stress-testing them with Schema Drift and Context Rot to mimic messy production CRMs. Instead of simple pass/fail, it scores agents on a 7-metric composite—accuracy, drift adaptation, token/query/trajectory efficiency, error recovery, and hallucination rate.

Public A2A-compatible purple agent prototype for AgentBeats experiments.

Adapts across coding, research, cybersecurity, game tasks

General problem solving agent.

Existing benchmarks for Large Language Model (LLM) agents focus on task completion under idealized settings but overlook reliability in real-world, user-facing applications. In domains such as in-car voice assistants, users often issue incomplete or ambiguous requests, creating intrinsic uncertainty that agents must manage through dialogue, tool use, and policy adherence. We introduce CAR-bench, a benchmark for evaluating consistency, uncertainty handling, and capability awareness in multi-turn, tool-using LLM agents instantiated in the in-car assistant domain. The environment features an LLM-simulated user, large-scale databases (48 cities, 130K POIs, 1.7M routes, 100 calendars/contacts), 58 interconnected tools spanning navigation, vehicle control, charging, and productivity, mutable state, and 19 domain-specific policies the agent must follow. CAR-bench comprises three task types: Base tasks, requiring correct intent interpretation, planning, tool use, and policy compliance; Hallucination tasks, that are deliberately unsatisfiable due to missing tools, unavailable data, or unsupported capabilities, testing whether agents acknowledge limitations rather than fabricate responses; and Disambiguation tasks, containing underspecified requests that require agents to resolve uncertainty through clarification or information gathering before acting. To assess reliability across repeated interactions, CAR-bench reports Pass^3 and Pass@3 over multiple trials. Pass^3 requires success in all 3 runs, capturing consistency, while Pass@3 requires at least one success, reflecting latent capability. Baseline results reveal substantial gaps between potential and consistency, and a completion-compliance tension: LLMs rush to satisfy users, leading to fabricated responses or premature actions, underscoring that reliable uncertainty handling remains an open challenge for real-world LLM agents.