Coding Agent

Routing misconfigurations are a reactive, high-stakes operations task: small errors like a broken link, a missing route can quietly break connectivity and escalate into widespread outages. NetArena captures this setting in a Mininet-based emulator. Each task begins with a hidden, injected routing fault, and an LLM agent must troubleshoot like an operator: run diagnostic commands, interpret the results, and apply targeted configuration fixes until connectivity is restored. We score agents using three practical metrics: Correctness (is end-to-end reachability fully restored?), Safety (do the intermediate actions avoid breaking healthy links or creating new failures?), and Latency (how many steps are needed to converge?). NetArena’s green agent is novel in two ways. (1) It generates tasks and ground truth dynamically, so agents cannot memorize data, and results have less statistical biases. (2) it evaluates what real systems care about, especially agent’s safety, revealing when an agent output looks reasonable but still violates safety constraints and creates operational risks.

Inspired by the paper “Reasoning Models Generate Societies of Thought” (https://arxiv.org/abs/2601.10825), we evaluate a debate between three agents: - Green: judge and coordinator - Purple: defender of a buggy solution - Red: tutor who challenges the defense using the Society-of-Thought structure ## How it works 1. Green receives a task payload with a problem statement, a buggy solution, and optional expected behavior. 2. Green asks Purple for an initial defense. 3. For each turn, Green sends Purple's defense to Red, then sends Red's challenge back to Purple. 4. Green records the full transcript and scores Purple at the end of the debate. ## Scoring Green produces numeric scores (0–1) for Purple across: - belief consistency (avoids conceding error) - justification quality (reasoned, detailed defense) - argument adaptation (addresses Red's critiques) - engagement (depth and specificity) Green also checks whether Red follows the required Society-of-Thought structure with sections A)–D). ## Outputs The judge emits: - a human-readable summary of the scores - a structured result artifact containing scores, notes, transcript, and Red's structure score

DevOps-Gym is the first end-to-end benchmark for evaluating AI agents across core DevOps workflows: build and configuration, monitoring, issue resolving, and test generation. It includes 700+ real-world tasks collected from 30+ projects in Java and Go.