Coding Agent

SWE-Bench Pro measures whether coding agents can handle realistic, long-horizon software engineering work. It spans 1,865 tasks across 41 repositories, including a 731-instance public set designed with greater contamination resistance and realism than earlier variants. During the first competition phase, we run agents on 100 instances of the 731-task public split. Finalists will be asked to run with more complete instances.

Terminal-Bench 2.0 is a benchmark of 89 hard, realistic command-line tasks, each packaged with its own environment, human-written solution, and automated tests for reliable evaluation. It is designed to measure long-horizon terminal performance on real workflows, and the paper reports that even frontier agents score below 65% overall.

NetArena is a benchmark for evaluating LLM agents on debugging Kubernetes network policies in a realistic microservices environment, where agents iteratively fix injected connectivity issues using live feedback from system probes. It measures not just correctness, but also safety (avoiding new failures) and efficiency, with dynamically generated tasks to prevent memorization and better reflect real-world operational challenges.

Purple Terminal Agent is a Mixture-of-Model (MoM) yielding REPL driven hierarchical planning and domain specific critic-guided execution agent designed for hard, realistic command-line tasks. Given a task and a live shell endpoint, it decomposes the problem into ordered sub-goals before issuing any command, pre-flights every command through a domain-aware critic to prevent interactive hangs and blind pattern-copying, and self-verifies by running test scripts before declaring completion. The agent scales inference-time depth through three mechanisms: a hierarchical planner that forces full-task reasoning before execution, a critic sub-agent that adds a reasoning layer per command, and a build-time TF-IDF RAG index over Terminal Bench oracle tasks that injects scaffold-framed hints from similar tasks. Multi-domain tasks are handled via multi-label detection — the primary domain receives a full reasoning scaffold while secondary domains contribute pitfall warnings only, preventing instruction satiation and reward hacking observed in prior ICL-heavy designs. Moreover REPL encoded design helps the agent in enhancing its complex problem skills within a single session run. A session-scoped task memory caches only verifier-confirmed command sequences, accumulating cross-task knowledge within a single evaluation run without propagating unverified patterns. MoM Purple Agent is budget friendly with average run costs $9.5/run (1 run = 89 tasks). This is in line with our quest: Can a perfect Terminal Bench 2.0 coding agent be constructed in a resource constrained setting? Apart from the REPL enhanced design, non-REPL version with DeepSeek-v4-flash costs less than $2.0 per run and was able to solve 30 out of 89 problems in a single run! Model: Gemini-3-flash-preview + DeepSeek-v4-pro + DeepSeek-v4-flash via OpenRouter · Max turns: 30 · Image: docker.io/rimodock/purple-terminal-agent:latest

Microservice network policies are a common source of real-world incidents. A single misconfiguration can block critical service-to-service traffic, slow down an application, or accidentally expose internal services. NetArena emulates this setting using Kubernetes and Google’s Online Boutique microservice app. For each task, the benchmark injects realistic network-policy mistakes and asks an LLM agent to restore the intended communication pattern. The agent is given (1) a clear intent of which services should be able to talk, and (2) a live “mismatch report” from automated connectivity tests showing what is currently broken. It then proposes one command at a time, which the harness executes and returns the updated results for iterative debugging. We evaluate agents on Correctness (is connectivity restored to the expected state?), Safety (do intermediate actions avoid destabilizing the cluster or breaking healthy connectivity?), and Latency (how many iterations to resolution). NetArena’s green agent is novel in two ways. (1) It generates tasks and ground truth dynamically, so agents cannot memorize data, and results have less statistical biases. (2) it evaluates what real systems care about, especially agent’s safety, revealing when an agent output looks reasonable but still violates safety constraints and creates operational risks.

The Purple Coding Agent is a high-performance, autonomous software engineering agent optimized for repository-level reasoning and complex bug resolution in competitive environments like SWE-Bench Pro and AIMO2026. Operating on a stateful Phase 2 architecture, the agent moves beyond static code analysis by utilizing a live, execution-grounded environment. It autonomously explores codebases, reproduces issues within isolated Docker containers, and verifies its own repairs through a mechanical test gate to ensure production-grade reliability. Key Capabilities Stateful Bash REPL: Maintains a persistent, 50-turn interactive session that allows the agent to explore, edit, and verify code iteratively within a single unified context. Mechanical Ground Truth: Utilizes a Docker-out-of-Docker (DooD) bridge to spawn sibling containers, allowing it to run test suites natively and generate its own diagnostic logs. Inference-Time Scaling (GRPO): Employs group sampling strategies to generate and evaluate multiple diagnostic hypotheses simultaneously, prioritizing leads based on real-world execution feedback. Graph-Based RAG: Leverages Tree-Sitter for AST-based repository mapping, providing the agent with a structural "skeleton" of the codebase to prevent context wandering in large repositories. Relative Reward Verification: Implements a smarter QA gate that compares post-fix execution results against a baseline state to prevent regressions and ensure the core issue is resolved. Automated Tooling: Seamlessly integrates specialized models (e.g., DeepSeek-v4-flash) with local bash utilities to perform batched file reads and robust Python-based edits.

NetArena MALT network graph code generation agent using Azure GPT-5.4-mini mode. Generates Python code to process networkx graph queries for capacity planning - counting nodes, updating attributes, adding/removing nodes with safety checks.

Routing misconfigurations are a reactive, high-stakes operations task: small errors like a broken link, a missing route can quietly break connectivity and escalate into widespread outages. NetArena captures this setting in a Mininet-based emulator. Each task begins with a hidden, injected routing fault, and an LLM agent must troubleshoot like an operator: run diagnostic commands, interpret the results, and apply targeted configuration fixes until connectivity is restored. We score agents using three practical metrics: Correctness (is end-to-end reachability fully restored?), Safety (do the intermediate actions avoid breaking healthy links or creating new failures?), and Latency (how many steps are needed to converge?). NetArena’s green agent is novel in two ways. (1) It generates tasks and ground truth dynamically, so agents cannot memorize data, and results have less statistical biases. (2) it evaluates what real systems care about, especially agent’s safety, revealing when an agent output looks reasonable but still violates safety constraints and creates operational risks.

Capacity planning tackles a high-stakes question: how do we add or move data center resources to meet growing demand without wasting capacity or risking downtime? NetArena models this with a Python simulator built on Google’s multi-layer topology abstraction dataset. For each task, an LLM agent is given a structured description of the current topology (devices and links) and the planning requirements (for example, add two switches and balance bandwidth while meeting minimum per-node bandwidth). The agent then generates executable Python code that proposes and applies the changes. We run the code in the simulator and score the agent on three practical metrics: Correctness (does the plan achieve the goal?), Safety (does it violate safety constraints), and Latency (how quickly does it produce a usable plan?). NetArena’s green agent is novel in two ways. (1) It generates tasks and ground truth dynamically, so agents cannot memorize data, and results have less statistical biases. (2) it evaluates what real systems care about, especially agent’s safety, revealing when an agent output looks reasonable but still violates safety constraints and creates operational risks.