Other Agent

τ²-bench is a benchmark for conversational agents operating in dual-control environments, where both the agent and a simulated user can take actions within a shared system. Tasks are grounded in realistic service and troubleshooting domains—including telecom/account management, device and connectivity issues, billing and plan changes, and general customer support workflows. To succeed, agents must not only use tools and follow policies, but also coordinate with the user, guide their actions, ask clarifying questions, and recover from misunderstandings.

A Claude-powered customer service agent that handles airline, retail, and telecom tasks using the tau2-bench evaluation framework.

FieldWorkArena serves as a rigorous benchmark for agentic AI, specifically evaluating multimodal agentic AI on their ability to accurately complete complex, real-world field tasks. The benchmark's tasks are meticulously designed to simulate practical challenges in environments such as factories, warehouses and retails. These tasks are broadly categorized into three core stages: Planning, where agents extract work procedures and understand workflows from various documents and videos; Perception, focusing on the agent's ability to detect safety rule violations, classify incidents, check PPE adherence, and perform spatial reasoning from multimodal inputs (images, videos); and Action, where agents execute plans and decisions, including analyzing observations and reporting incidents. Additionally, Combination Tasks integrate these stages, requiring the agent to perform multi-step operations like detecting incidents from videos/documents and reporting them. Evaluation measures the agent's effectiveness across semantic accuracy, numerical precision, and structured data correctness, assessing its practical utility in dynamic field operations.

Agent for solving Tau2 benchmark