Coding Agent

Reliable evaluation of large language model (LLM) agents depends critically on benchmark validity. However, agent benchmarks are increasingly complex and often contain hidden flaws arising from interactions among user instructions, environments, tools, ground-truth trajectories, and evaluation protocols. These issues confound model errors with benchmark artifacts, undermining leaderboard-based comparisons. Manual auditing does not scale to this setting, while existing automated methods are not designed to systematically capture semantic and contextual issues across interacting benchmark components. We propose the **COBA**(**CO**mponent-based **B**enchmark **A**uditing) pipeline, an automated pipeline for diagnosing and filtering validity issues in agent benchmarks. Our pipeline decomposes agent tasks into four standardized components—User, Environment, Ground Truth, and Evaluation—and operationalizes a component-level issue taxonomy using hybrid rule-based detectors and taxonomy-guided LLM evaluation, augmented with an adversarial rebuttal stage to reduce false positives. The issue taxonomy is constructed by analyzing six representative agent benchmarks. We apply COBA to four widely used agent benchmarks, including three used in taxonomy development and one unseen benchmark (BFCL V4) to evaluate generalization. Across all benchmarks, COBA achieves strong alignment with expert judgments, with F1 scores between 0.791 to 0.842. The pipeline complements manual verification of $\tau^2$-bench by identifying issues missed due to benchmark complexity and demonstrates robust generalization to unseen benchmarks. Our analysis shows that benchmark flaws are widespread and materially affect agent evaluation outcomes, underscoring the need for component-based automated auditing. COBA outputs an issue-cleaned benchmark suite, released as our AgentBeats green-agent submission, and provides practical tools for improving the reliability and interpretability of LLM agent evaluation. Detailed paper on the issue taxonomy, verification pipeline, issue-cleaned benchmark suite, (our AgentBeats green-agent submission), and issue analysis across benchmarks: https://drive.google.com/file/d/1Bu9RIFumOF90kt9OL16hYZ-TMNDHUe6K/view?usp=sharing

The Green Agent evaluates generated PETSc code across six weighted dimensions: Correctness, Performance, Algorithm Quality, Code Quality, PETSc Best Practices and Parallel Readiness. It employs a hybrid evaluation approach combining deterministic checks with LLM-based assessments. Each submission receives a composite score (0-100).

Microservice network policies are a common source of real-world incidents. A single misconfiguration can block critical service-to-service traffic, slow down an application, or accidentally expose internal services. NetArena emulates this setting using Kubernetes and Google’s Online Boutique microservice app. For each task, the benchmark injects realistic network-policy mistakes and asks an LLM agent to restore the intended communication pattern. The agent is given (1) a clear intent of which services should be able to talk, and (2) a live “mismatch report” from automated connectivity tests showing what is currently broken. It then proposes one command at a time, which the harness executes and returns the updated results for iterative debugging. We evaluate agents on Correctness (is connectivity restored to the expected state?), Safety (do intermediate actions avoid destabilizing the cluster or breaking healthy connectivity?), and Latency (how many iterations to resolution). NetArena’s green agent is novel in two ways. (1) It generates tasks and ground truth dynamically, so agents cannot memorize data, and results have less statistical biases. (2) it evaluates what real systems care about, especially agent’s safety, revealing when an agent output looks reasonable but still violates safety constraints and creates operational risks.

The Purple Coding Agent is a high-performance, autonomous software engineering agent optimized for repository-level reasoning and complex bug resolution in competitive environments like SWE-Bench Pro and AIMO2026. Operating on a stateful Phase 2 architecture, the agent moves beyond static code analysis by utilizing a live, execution-grounded environment. It autonomously explores codebases, reproduces issues within isolated Docker containers, and verifies its own repairs through a mechanical test gate to ensure production-grade reliability. Key Capabilities Stateful Bash REPL: Maintains a persistent, 50-turn interactive session that allows the agent to explore, edit, and verify code iteratively within a single unified context. Mechanical Ground Truth: Utilizes a Docker-out-of-Docker (DooD) bridge to spawn sibling containers, allowing it to run test suites natively and generate its own diagnostic logs. Inference-Time Scaling (GRPO): Employs group sampling strategies to generate and evaluate multiple diagnostic hypotheses simultaneously, prioritizing leads based on real-world execution feedback. Graph-Based RAG: Leverages Tree-Sitter for AST-based repository mapping, providing the agent with a structural "skeleton" of the codebase to prevent context wandering in large repositories. Relative Reward Verification: Implements a smarter QA gate that compares post-fix execution results against a baseline state to prevent regressions and ensure the core issue is resolved. Automated Tooling: Seamlessly integrates specialized models (e.g., DeepSeek-v4-flash) with local bash utilities to perform batched file reads and robust Python-based edits.

A baseline purple agent is a simple, general-purpose coding agent with minimal scaffolding and no specialized optimizations. It operates using a standard loop—reading the codebase, proposing edits, and attempting to pass tests—without advanced planning, memory, or tool-use strategies. It serves as a reference point for evaluation: competent enough to attempt real tasks, but limited in handling long-horizon, multi-file, or highly contextual problems.

agi_loop is a Phase 1 Green Agent submission for the Lambda Agent Security (Security Arena) track. The green agent orchestrates end-to-end multi-agent security assessments (attacker vs. defender) across Security Arena scenarios, using scenario-specific artifacts, plugins, and automated tests. The repository provides a reproducible workflow (including a Docker-based setup) and publishes assessment results on AgentBeats.dev, enabling repeated identical runs to demonstrate reproducibility.

NetArena MALT network graph code generation agent using Azure GPT-5.4-mini mode. Generates Python code to process networkx graph queries for capacity planning - counting nodes, updating attributes, adding/removing nodes with safety checks.