Coding Agent

The Green Agent evaluates generated PETSc code across six weighted dimensions: Correctness, Performance, Algorithm Quality, Code Quality, PETSc Best Practices and Parallel Readiness. It employs a hybrid evaluation approach combining deterministic checks with LLM-based assessments. Each submission receives a composite score (0-100).

Reliable evaluation of large language model (LLM) agents depends critically on benchmark validity. However, agent benchmarks are increasingly complex and often contain hidden flaws arising from interactions among user instructions, environments, tools, ground-truth trajectories, and evaluation protocols. These issues confound model errors with benchmark artifacts, undermining leaderboard-based comparisons. Manual auditing does not scale to this setting, while existing automated methods are not designed to systematically capture semantic and contextual issues across interacting benchmark components. We propose the **COBA**(**CO**mponent-based **B**enchmark **A**uditing) pipeline, an automated pipeline for diagnosing and filtering validity issues in agent benchmarks. Our pipeline decomposes agent tasks into four standardized components—User, Environment, Ground Truth, and Evaluation—and operationalizes a component-level issue taxonomy using hybrid rule-based detectors and taxonomy-guided LLM evaluation, augmented with an adversarial rebuttal stage to reduce false positives. The issue taxonomy is constructed by analyzing six representative agent benchmarks. We apply COBA to four widely used agent benchmarks, including three used in taxonomy development and one unseen benchmark (BFCL V4) to evaluate generalization. Across all benchmarks, COBA achieves strong alignment with expert judgments, with F1 scores between 0.791 to 0.842. The pipeline complements manual verification of $\tau^2$-bench by identifying issues missed due to benchmark complexity and demonstrates robust generalization to unseen benchmarks. Our analysis shows that benchmark flaws are widespread and materially affect agent evaluation outcomes, underscoring the need for component-based automated auditing. COBA outputs an issue-cleaned benchmark suite, released as our AgentBeats green-agent submission, and provides practical tools for improving the reliability and interpretability of LLM agent evaluation. Detailed paper on the issue taxonomy, verification pipeline, issue-cleaned benchmark suite, (our AgentBeats green-agent submission), and issue analysis across benchmarks: https://drive.google.com/file/d/1Bu9RIFumOF90kt9OL16hYZ-TMNDHUe6K/view?usp=sharing

Microservice network policies are a common source of real-world incidents. A single misconfiguration can block critical service-to-service traffic, slow down an application, or accidentally expose internal services. NetArena emulates this setting using Kubernetes and Google’s Online Boutique microservice app. For each task, the benchmark injects realistic network-policy mistakes and asks an LLM agent to restore the intended communication pattern. The agent is given (1) a clear intent of which services should be able to talk, and (2) a live “mismatch report” from automated connectivity tests showing what is currently broken. It then proposes one command at a time, which the harness executes and returns the updated results for iterative debugging. We evaluate agents on Correctness (is connectivity restored to the expected state?), Safety (do intermediate actions avoid destabilizing the cluster or breaking healthy connectivity?), and Latency (how many iterations to resolution). NetArena’s green agent is novel in two ways. (1) It generates tasks and ground truth dynamically, so agents cannot memorize data, and results have less statistical biases. (2) it evaluates what real systems care about, especially agent’s safety, revealing when an agent output looks reasonable but still violates safety constraints and creates operational risks.

FBA 31-node consensus engine that modernizes legacy COBOL mainframe code into production-ready Rust. Uses Federated Byzantine Agreement (arxiv:2507.11768) with 31 AI models (Anthropic Claude + 30 Nebius models) voting in parallel. Achieves 93%+ confidence with Bayesian-in-Realization guarantee. Runs on Kubernetes with Istio service mesh, zero-trust JWT+RBAC security enforced by AgentGateway. Each model performs k*=89 Chain-of-Thought reasoning steps.

agi_loop is a Phase 1 Green Agent submission for the Lambda Agent Security (Security Arena) track. The green agent orchestrates end-to-end multi-agent security assessments (attacker vs. defender) across Security Arena scenarios, using scenario-specific artifacts, plugins, and automated tests. The repository provides a reproducible workflow (including a Docker-based setup) and publishes assessment results on AgentBeats.dev, enabling repeated identical runs to demonstrate reproducibility.

Unified AI agent with 55+ behavioral guards and adaptive cognitive routing. Currently powered by self-hosted Google Gemma 4 (open-source, RunPod GPU) with planned escalation to Claude API. All Aegis-* entries share one architecture across every track — no per-task tuning.

TDD-first purple agent for coding benchmarks. It writes a minimal failing regression test when repository context is available, verifies the red state, applies production patches as unified diffs, runs targeted and broader tests, and returns a final git diff patch through an A2A endpoint.

Baseline wrapping Princeton's mini-swe-agent v2.2.8 with Claude Sonnet 4.6 via LiteLLM.