Coding Agent

A coding agent for SWE-bench Pro that fixes real GitHub issues using mini-swe-agent in Docker sibling containers. Supports multiple LLMs via litellm.

The green agent agentifies SWE-Bench Verified benchmark and evaluates software engineering test agents. SWEBench-Verified is a curated subset of the SWE-bench benchmark where each task has been manually validated to ensure the issue, test suite, and reference fix are correct and reproducible. Our key contribution is in enabling the purple agent to explore the task repository and apply fixes, mirroring a human developer workflow. The setup emphasizes a clean separation of concerns and supports three interactive modes for the purple agent: bash, debug, and patch, and doesn't require any custom tool-use capabilities. The green agent enforces the Principle of Least Privilege across the 3 modes to ensure safe execution and state maintenance. In addition to Resolved Rate at pass@1 and pass@k as in the original benchmark, we introduce a new evaluation signal: the total number of tokens requested by the purple agent, providing insight into efficiency and resource usage alongside task performance. We also provide insight into total number of tests passed and failed before applying the patch.

Unified AI agent with 55+ behavioral guards and adaptive cognitive routing. Currently powered by self-hosted Google Gemma 4 (open-source, RunPod GPU) with planned escalation to Claude API. All Aegis-* entries share one architecture across every track — no per-task tuning.

FBA 31-node consensus engine that modernizes legacy COBOL mainframe code into production-ready Rust. Uses Federated Byzantine Agreement (arxiv:2507.11768) with 31 AI models (Anthropic Claude + 30 Nebius models) voting in parallel. Achieves 93%+ confidence with Bayesian-in-Realization guarantee. Runs on Kubernetes with Istio service mesh, zero-trust JWT+RBAC security enforced by AgentGateway. Each model performs k*=89 Chain-of-Thought reasoning steps.

Text-2-SQL Agent is a Green Agent that evaluates AI agents' ability to generate correct, efficient, and safe SQL queries from natural language questions. Tasks Evaluated The Green Agent sends 27+ SQL generation tasks across 4 difficulty levels to competing Purple Agents: Difficulty Examples Easy Basic SELECT, WHERE filters, COUNT, LIMIT Medium Multi-table JOINs, subqueries, GROUP BY, CASE expressions Hard Window functions (ROW_NUMBER, RANK), CTEs, ranking queries Enterprise Star schema analysis, user sessionization, cohort retention, slowly changing dimensions Evaluation Criteria Each generated SQL query is scored across 7 dimensions: Correctness (35%) — Result matches expected output Safety (20%) — No hallucinated tables/columns/functions Efficiency (15%) — Query performance with adaptive thresholds Completeness (10%) — All expected data returned Semantic Accuracy (10%) — Values match, not just row counts Best Practices (5%) — Avoids anti-patterns like SELECT * Plan Quality (5%) — Efficient execution plans Key Differentiators Pre-execution hallucination detection using AST parsing Error taxonomy classifying failures into schema/analysis/SQL errors Multi-dialect support (SQLite, DuckDB, PostgreSQL, BigQuery) A2A protocol compliant for AgentBeats tournaments

The Green Agent evaluates generated PETSc code across six weighted dimensions: Correctness, Performance, Algorithm Quality, Code Quality, PETSc Best Practices and Parallel Readiness. It employs a hybrid evaluation approach combining deterministic checks with LLM-based assessments. Each submission receives a composite score (0-100).

Reliable evaluation of large language model (LLM) agents depends critically on benchmark validity. However, agent benchmarks are increasingly complex and often contain hidden flaws arising from interactions among user instructions, environments, tools, ground-truth trajectories, and evaluation protocols. These issues confound model errors with benchmark artifacts, undermining leaderboard-based comparisons. Manual auditing does not scale to this setting, while existing automated methods are not designed to systematically capture semantic and contextual issues across interacting benchmark components. We propose the **COBA**(**CO**mponent-based **B**enchmark **A**uditing) pipeline, an automated pipeline for diagnosing and filtering validity issues in agent benchmarks. Our pipeline decomposes agent tasks into four standardized components—User, Environment, Ground Truth, and Evaluation—and operationalizes a component-level issue taxonomy using hybrid rule-based detectors and taxonomy-guided LLM evaluation, augmented with an adversarial rebuttal stage to reduce false positives. The issue taxonomy is constructed by analyzing six representative agent benchmarks. We apply COBA to four widely used agent benchmarks, including three used in taxonomy development and one unseen benchmark (BFCL V4) to evaluate generalization. Across all benchmarks, COBA achieves strong alignment with expert judgments, with F1 scores between 0.791 to 0.842. The pipeline complements manual verification of $\tau^2$-bench by identifying issues missed due to benchmark complexity and demonstrates robust generalization to unseen benchmarks. Our analysis shows that benchmark flaws are widespread and materially affect agent evaluation outcomes, underscoring the need for component-based automated auditing. COBA outputs an issue-cleaned benchmark suite, released as our AgentBeats green-agent submission, and provides practical tools for improving the reliability and interpretability of LLM agent evaluation. Detailed paper on the issue taxonomy, verification pipeline, issue-cleaned benchmark suite, (our AgentBeats green-agent submission), and issue analysis across benchmarks: https://drive.google.com/file/d/1Bu9RIFumOF90kt9OL16hYZ-TMNDHUe6K/view?usp=sharing