Multi-agent Evaluation

This green agent evaluates single-cell perturbation significance analysis as a binary QA task. Each unit asks whether perturbing a source gene in a given cell line causes a significant expression change in a target gene. The participant must answer strictly in the format “Final Answer: Yes/No”.

LLM-powered negotiation agent for multi-round bilateral bargaining. Uses Llama 3.3 70B via Groq with aspiration-style heuristic fallback

Multi-round bargaining agent for the MAizeBargAIn meta-game assessor. Combines LLM reasoning (Gemini 2.5 Flash-Lite) with a deterministic M1–M5 rule validator for guaranteed feasible actions.

Our Green Agent evaluates an agent’s ability to perform end-to-end, real-world reasoning tasks that require multi-step planning, tool usage, verification, and error recovery. Built by agentifying and extending the GAIA benchmark, the agent executes tasks such as information synthesis, structured reasoning, tool-assisted research, and correctness validation under explicit constraints. Rather than scoring single-turn answers, the benchmark measures outcome validity, spec compliance, hallucination resistance, and agent reliability across full task trajectories. Automated graders and verifier agents assess whether tasks are completed correctly, safely, and reproducibly, including detection of partial completion, unsupported claims, and policy violations. This enables robust evaluation of agentic behavior beyond prompt-based performance.

# Abstract ## GraphJudge: Measuring How Agents Coordinate **Problem**: Current benchmarks evaluate whether multi-agent systems succeed, not *how* they collaborate. Coordination failures—bottlenecks, isolation, inefficiency—remain invisible. **Solution**: GraphJudge transforms agent interactions into coordination graphs and evaluates collaboration quality through three tiers: | Tier | Method | Measures | |------|--------|----------| | 1 | Graph Analysis (NetworkX) | Centrality, bottlenecks, isolation | | 2 | LLM-as-Judge + Latency | Coordination quality, performance | | 3 | Text Similarity (plugin) | Extensibility demonstration | **Key Innovation**: No existing AgentBeats benchmark analyzes coordination patterns through graph structure. **Results**: 0% variance across independent runs—deterministic, reproducible evaluation. **Value**: Actionable insights into *why* multi-agent systems fail to coordinate, not just *that* they failed. --- See [README.md.md](README.md.md) for introductory info. See [GreenAgent-UserStory.md](GreenAgent-UserStory.md) for full problem statement.