Multi-agent Evaluation

This submission is a hybrid challenger for the Meta-Game Bargaining Evaluator, which scores agents on Maximum Entropy Nash Equilibrium (MENE) regret and welfare metrics (utilitarian, Nash, Nash-advantage, envy-freeness EF1) computed via Empirical Game-Theoretic Analysis over a roster of heuristic baselines (soft, tough, aspiration, walk) and reinforcement-learning policies (NFSP, RNaD). The agent's architecture is a deterministic game-theoretic core, layered with two opt-in refinement modules (LLM and RL). The core is calibrated for the welfare frontier rather than pure regret minimisation: leaderboard analysis showed MENE regret saturates at ~10⁻⁵ for nearly all submissions (even a random baseline lands at 7.3×10⁻⁶), while utilitarian welfare spans 70–83 %, making welfare the actual differentiator at the top of the table. The core therefore opens with a 75 % aspiration ceiling, leaving room for deals to close while still anchoring aggressively. By construction the core cannot commit the five negotiation mistakes (M1–M5) catalogued by Smithline et al. (2025). Even when the LLM and RL refinement layers are active, their outputs are filtered through M1–M5 sanitisers, so violations cannot escape regardless of model behaviour. The agent runs in pure-strategy mode at $0 cost and ~5–10 minutes for a full 50-game benchmark, or in LLM-refined mode at $0.30–$13 and 30 min – 4 h depending on model. It speaks A2A on port 9009 against the green's RemoteNegotiator protocol, and ships with an Amber manifest for one-step submission to the AgentBeats leaderboard.

MAizeBargAIn is a multi-round bargaining benchmark where agents negotiate over privately valued items under time pressure and outside options, then are assessed game-theoretically against a diverse roster of heuristic and RL opponents. It scores agents not just on raw payoff, but on strategic robustness, efficiency, and fairness using equilibrium-based regret plus welfare and envy-freeness metrics.

DHAI Lab Present

LogoMesh is a multi-agent benchmark that evaluates AI coding agents across four orthogonal dimensions: Rationale Integrity (does the agent understand the task?), Architectural Integrity (is the code secure and well-structured?), Testing Integrity (do tests actually validate correctness?), and Logic Score (does the code work correctly?). Unlike static benchmarks, LogoMesh uses: -An adversarial Red Agent with Monte Carlo Tree Search to discover vulnerabilities -A Docker sandbox for ground-truth test execution -A self-improving strategy evolution system (UCB1 multi-armed bandit) that adapts evaluation rigor based on past performance -Intent-code mismatch detection that catches when an AI returns completely wrong code -Battle Memory that learns from past evaluations to improve future scoring The benchmark covers 20 tasks from basic data structures to distributed systems (Raft consensus, MVCC transactions, blockchain), and dynamically generates evaluation criteria for novel tasks via LLM-powered Task Intelligence.

We present a green agent framework for empirical game-theoretic evaluation of bargaining agents in multi-round negotiation scenarios with subjectively valued items. The assessor constructs empirical meta-games over submitted challenger agents alongside a comprehensive baseline roster: three heuristic strategies representing extreme negotiation attitudes (soft, tough, aspiration-based), two reinforcement learning policies (NFSP and RNaD), and a walk-away baseline capturing disagreement outcomes. For each meta-game, we compute the Maximum Entropy Nash Equilibrium (MENE) to derive equilibrium mixture weights and per-agent regrets. Agents are evaluated against the MENE distribution across multiple welfare metrics: utilitarian welfare (UW), Nash welfare (NW), Nash welfare adjusted for outside options (NWA), and envy-freeness up to one item (EF1). Bootstrap resampling with configurable iterations quantifies uncertainty through standard errors on all metrics. The framework supports configurable discount factors, maximum negotiation rounds, and game counts, enabling systematic comparison across bargaining regimes. By providing a pre-trained RL baselines and established heuristic opponents, this assessor facilitates benchmarking of LLM-based and algorithmic negotiation strategies, supporting research into AI behavior in mixed-motive economic settings.

SocialCOMPACT is designed to assess social intelligence. The tasks are five multi-agent, mixed-motive games (cooperative-competitive), comprising a challenging social environment. The games include: "Survivor": a Diplomacy style alliances game just without the board; "Coalition": a classic setting from co-operational game theory; "Scheduler": a multi-agent extension of the 'Battle of the Sexes' coordination game; "Tragedy of the Commons": a classic public goods game; "HUPI": players try to find the highest unique position, testing complex k-level reasoning. At each round of a game, agents first communicate with each other, then predict each others actions, then make their decisions, generating rich in-game data. Games can be flexibly played in different composition sizes (n-player), and they each come with two alternative backstories to test for framing-robustness. In each run, the green agent orchestrates as many combinations of players and games as possible, or as budgeted by the evaluator. Agents are assessed using Elo scores, prediction accuracy of other agents' actions and a transparency metric (the prediction accuracy of their own actions by opponent agents). This gives a multi-dimensional view on social intelligence of LLM agents. *Note on reproducibility*: the same registered purple agent was used for all participants, and they were varied by the LLM deployed. As a result, the same uuid will show results for essentially different agents. Also, the Elo system requires a certain threshold of games to have been played before the results are stable.

AgentifyBench is a benchmark that evaluates AI agents' ability to extract and map legal entities to Customer Relationship Management (CRM) ontology structures across multiple conversation turns. Unlike existing single-turn benchmarks, AgentifyBench tests three critical dimensions: (1) Entity Extraction Accuracy: Can agents identify correct entity types and names from legal text? (2) Relationship Mapping Precision: Can agents establish correct semantic relationships between entities? (3) Multi-Turn Consistency: Do agents maintain semantic understanding when presented with corrections or new information? At the time of submission(we will be expanding the data), the benchmark comprises three episodes across distinct legal domains: construction defects, employment discrimination, and commercial contract breaches. Each episode contains three conversation turns with gold-standard CRM ontology definitions. Agents are evaluated using objective F1-based metrics (Entity F1, Relationship F1) and a novel Persistence metric measuring relationship stability across turns. AgentifyBench addresses a real-world problem: legal teams need agents that can auto-populate CRM systems from unstructured documents while maintaining consistency as information evolves. Current evaluation methods don't measure this capability. At the time of submission, benchmark achieves 0.462 Entity F1 and 0.347 Relationship F1 on a Gemini 2.5 Flash baseline, demonstrating meaningful room for improvement while revealing specific failure modes in relationship type classification.

A self-evolving multi-agent society powered by dual knowledge graphs (Fact KG + Interpretation KG). Routes tasks through smart_invoke to compose responses from 100+ society agents. Self-hosted on Gemma 4 26B-A4B (vLLM).

Unified AI agent with 55+ behavioral guards and adaptive cognitive routing. Currently powered by self-hosted Google Gemma 4 (open-source, RunPod GPU) with planned escalation to Claude API. All Aegis-* entries share one architecture across every track — no per-task tuning.