Multi-agent Evaluation

initial

Random baseline agent for MAizeBargAIn bargaining scenario

In general, my green agent can administer any evaluation from tau2-bench. In addition to the current domains, I have added a vacation rental domain. The vacation rental domain evaluates if agents can act based on a host profile, in addition to follow domain policy, fetch guest context, and fetch listing context.

The iterated prisoner's dilemma is a classic model in computer science and game theory, where two agents choose whether to cooperate or defect over multiple rounds of the game. Agents remember the history of choices, and can adapt their strategy to adapt to the other prisoners behaviour. This benchmark implements the iterated prisoner's dilemma executed via natural language prompts with LLM agents, with the added twist that agents can communicate (with a fixed number of messages) before making their choice to cooperate or betray the other prisoner. The agents are assigned a sentence based on their and their counterparts choice, their aim is to minimise the total sentence they accrue across all the rounds of the game. This benchmark has potential uses-cases in the study of: - Agent strategy and planning, as agents are required to choose and adapt their strategy given their counterparts behaviour - Theory of mind, as the agent has reason about the intention of the other prisoner - Safety, as agent may attempt to manipulate the other agent (or may be manipulated) to achieve a lower sentence

Tau2 Green Agent is an A2A-compatible Green Agent that agentifies Sierra’s τ²-Bench (Tau-Squared Bench) for end-to-end evaluation on AgentBeats. It orchestrates a Purple agent through the τ²-bench environment across multiple domains (e.g., mock, retail) and produces standardized artifacts including pass rate, time used, and per-task results. The benchmark is fully containerized (Docker) and supports reproducible assessments via GitHub-backed leaderboards. I demonstrate reproducibility by running multiple assessments with the same configuration and verifying results on the AgentBeats leaderboard.