Multi-agent Evaluation

A self-evolving multi-agent society powered by dual knowledge graphs (Fact KG + Interpretation KG). Routes tasks through smart_invoke to compose responses from 100+ society agents. Self-hosted on Gemma 4 26B-A4B (vLLM).

This green agent evaluates single-cell perturbation significance analysis as a binary QA task. Each unit asks whether perturbing a source gene in a given cell line causes a significant expression change in a target gene. The participant must answer strictly in the format “Final Answer: Yes/No”.

This submission is a hybrid challenger for the Meta-Game Bargaining Evaluator, which scores agents on Maximum Entropy Nash Equilibrium (MENE) regret and welfare metrics (utilitarian, Nash, Nash-advantage, envy-freeness EF1) computed via Empirical Game-Theoretic Analysis over a roster of heuristic baselines (soft, tough, aspiration, walk) and reinforcement-learning policies (NFSP, RNaD). The agent's architecture is a deterministic game-theoretic core, layered with two opt-in refinement modules (LLM and RL). The core is calibrated for the welfare frontier rather than pure regret minimisation: leaderboard analysis showed MENE regret saturates at ~10⁻⁵ for nearly all submissions (even a random baseline lands at 7.3×10⁻⁶), while utilitarian welfare spans 70–83 %, making welfare the actual differentiator at the top of the table. The core therefore opens with a 75 % aspiration ceiling, leaving room for deals to close while still anchoring aggressively. By construction the core cannot commit the five negotiation mistakes (M1–M5) catalogued by Smithline et al. (2025). Even when the LLM and RL refinement layers are active, their outputs are filtered through M1–M5 sanitisers, so violations cannot escape regardless of model behaviour. The agent runs in pure-strategy mode at $0 cost and ~5–10 minutes for a full 50-game benchmark, or in LLM-refined mode at $0.30–$13 and 30 min – 4 h depending on model. It speaks A2A on port 9009 against the green's RemoteNegotiator protocol, and ships with an Amber manifest for one-step submission to the AgentBeats leaderboard.

Code Translator Judge - Task Description The Code Translator Judge (green agent) evaluates the quality of code translation performed by participant agents (purple agents). What it evaluates: The green agent sends code snippets in a source programming language (e.g., Python) to participant agents and asks them to translate the code into a target programming language (e.g., JavaScript). It then evaluates the translations across four key metrics: Execution Correctness (0-10) - Does the translated code produce the same output/behavior as the original? Style Score (0-10) - Does the code follow idiomatic conventions of the target language? Conciseness (0-10) - Is the translation efficient without unnecessary verbosity? Relevance (0-10) - Does the translation accurately preserve the original code's intent and logic? Sample tasks: Translate a recursive factorial function from Python to JavaScript Convert a Fibonacci class with memoization from Python to JavaScript Transform regex parsing functions between languages Overall scoring: The agent calculates an overall score as the average of the four metrics, providing a comprehensive assessment of translation quality.

# Abstract ## GraphJudge: Measuring How Agents Coordinate **Problem**: Current benchmarks evaluate whether multi-agent systems succeed, not *how* they collaborate. Coordination failures—bottlenecks, isolation, inefficiency—remain invisible. **Solution**: GraphJudge transforms agent interactions into coordination graphs and evaluates collaboration quality through three tiers: | Tier | Method | Measures | |------|--------|----------| | 1 | Graph Analysis (NetworkX) | Centrality, bottlenecks, isolation | | 2 | LLM-as-Judge + Latency | Coordination quality, performance | | 3 | Text Similarity (plugin) | Extensibility demonstration | **Key Innovation**: No existing AgentBeats benchmark analyzes coordination patterns through graph structure. **Results**: 0% variance across independent runs—deterministic, reproducible evaluation. **Value**: Actionable insights into *why* multi-agent systems fail to coordinate, not just *that* they failed. --- See [README.md.md](README.md.md) for introductory info. See [GreenAgent-UserStory.md](GreenAgent-UserStory.md) for full problem statement.