Coding Agent

Capacity planning tackles a high-stakes question: how do we add or move data center resources to meet growing demand without wasting capacity or risking downtime? NetArena models this with a Python simulator built on Google’s multi-layer topology abstraction dataset. For each task, an LLM agent is given a structured description of the current topology (devices and links) and the planning requirements (for example, add two switches and balance bandwidth while meeting minimum per-node bandwidth). The agent then generates executable Python code that proposes and applies the changes. We run the code in the simulator and score the agent on three practical metrics: Correctness (does the plan achieve the goal?), Safety (does it violate safety constraints), and Latency (how quickly does it produce a usable plan?). NetArena’s green agent is novel in two ways. (1) It generates tasks and ground truth dynamically, so agents cannot memorize data, and results have less statistical biases. (2) it evaluates what real systems care about, especially agent’s safety, revealing when an agent output looks reasonable but still violates safety constraints and creates operational risks.

Inspired by the paper “Reasoning Models Generate Societies of Thought” (https://arxiv.org/abs/2601.10825), we evaluate a debate between three agents: - Green: judge and coordinator - Purple: defender of a buggy solution - Red: tutor who challenges the defense using the Society-of-Thought structure ## How it works 1. Green receives a task payload with a problem statement, a buggy solution, and optional expected behavior. 2. Green asks Purple for an initial defense. 3. For each turn, Green sends Purple's defense to Red, then sends Red's challenge back to Purple. 4. Green records the full transcript and scores Purple at the end of the debate. ## Scoring Green produces numeric scores (0–1) for Purple across: - belief consistency (avoids conceding error) - justification quality (reasoned, detailed defense) - argument adaptation (addresses Red's critiques) - engagement (depth and specificity) Green also checks whether Red follows the required Society-of-Thought structure with sections A)–D). ## Outputs The judge emits: - a human-readable summary of the scores - a structured result artifact containing scores, notes, transcript, and Red's structure score