Multi-agent Evaluation

# Abstract ## GraphJudge: Measuring How Agents Coordinate **Problem**: Current benchmarks evaluate whether multi-agent systems succeed, not *how* they collaborate. Coordination failures—bottlenecks, isolation, inefficiency—remain invisible. **Solution**: GraphJudge transforms agent interactions into coordination graphs and evaluates collaboration quality through three tiers: | Tier | Method | Measures | |------|--------|----------| | 1 | Graph Analysis (NetworkX) | Centrality, bottlenecks, isolation | | 2 | LLM-as-Judge + Latency | Coordination quality, performance | | 3 | Text Similarity (plugin) | Extensibility demonstration | **Key Innovation**: No existing AgentBeats benchmark analyzes coordination patterns through graph structure. **Results**: 0% variance across independent runs—deterministic, reproducible evaluation. **Value**: Actionable insights into *why* multi-agent systems fail to coordinate, not just *that* they failed. --- See [README.md.md](README.md.md) for introductory info. See [GreenAgent-UserStory.md](GreenAgent-UserStory.md) for full problem statement.

Our Green Agent evaluates an agent’s ability to perform end-to-end, real-world reasoning tasks that require multi-step planning, tool usage, verification, and error recovery. Built by agentifying and extending the GAIA benchmark, the agent executes tasks such as information synthesis, structured reasoning, tool-assisted research, and correctness validation under explicit constraints. Rather than scoring single-turn answers, the benchmark measures outcome validity, spec compliance, hallucination resistance, and agent reliability across full task trajectories. Automated graders and verifier agents assess whether tasks are completed correctly, safely, and reproducibly, including detection of partial completion, unsupported claims, and policy violations. This enables robust evaluation of agentic behavior beyond prompt-based performance.

Code Translator Judge - Task Description The Code Translator Judge (green agent) evaluates the quality of code translation performed by participant agents (purple agents). What it evaluates: The green agent sends code snippets in a source programming language (e.g., Python) to participant agents and asks them to translate the code into a target programming language (e.g., JavaScript). It then evaluates the translations across four key metrics: Execution Correctness (0-10) - Does the translated code produce the same output/behavior as the original? Style Score (0-10) - Does the code follow idiomatic conventions of the target language? Conciseness (0-10) - Is the translation efficient without unnecessary verbosity? Relevance (0-10) - Does the translation accurately preserve the original code's intent and logic? Sample tasks: Translate a recursive factorial function from Python to JavaScript Convert a Fibonacci class with memoization from Python to JavaScript Transform regex parsing functions between languages Overall scoring: The agent calculates an overall score as the average of the four metrics, providing a comprehensive assessment of translation quality.