Coding Agent

Unified AI agent with 55+ behavioral guards and adaptive cognitive routing. Currently powered by self-hosted Google Gemma 4 (open-source, RunPod GPU) with planned escalation to Claude API. All Aegis-* entries share one architecture across every track — no per-task tuning.

Our green agent implements an A2A-compatible evaluator for the Data Analysis Benchmark (DABench), a benchmark designed to assess LLM-based agents on realistic data analysis tasks over CSV datasets. DABench defines end-to-end analytical questions that require agents to interpret data, perform transformations, and produce verifiable outputs, enabling systematic evaluation of data analysis capabilities (see DABench paper: https://arxiv.org/html/2401.05507v1). Within this setup, the green agent (1) loads and structures tasks from the DABench benchmark, (2) dispatches clear analytical instructions to a participating agent via the A2A protocol, and (3) evaluates the agent’s responses using an LLM-as-judge approach to assess correctness and completeness. The green agent focuses exclusively on orchestration and evaluation, while reasoning and code execution are fully handled by the participating agent.

AgentBeats-ready green agent for the ArchXBench RTL synthesis benchmark. The service exposes the A2A-compatible agent card plus task discovery and health endpoints, and evaluates Verilog submissions with Icarus Verilog (and optionally Yosys for PPA metrics).

SWE-Bench Verified Green Agent - Task Description The green agent is an automated evaluator designed to assess the software engineering capabilities of participant agents. It uses the SWE-Bench Verified dataset, which contains real-world GitHub issues and their corresponding fixes from popular open-source Python repositories. What It Evaluates The green agent measures how well a participant agent can understand a problem description, analyze a codebase, and produce a working patch that fixes the reported issue—all without introducing regressions to existing functionality. Evaluation Workflow The evaluation process consists of seven phases: Phase 1: Repository Cloning and Initial Checkout — The green agent clones the target repository from GitHub and checks out the environment_setup_commit. This is a specific commit where the environment is known to be stable for dependency installation. Phase 2: Environment Setup — Using an LLM-powered agentic loop, the green agent reads the repository's configuration files (like pyproject.toml or setup.py) to identify the required Python version. It then installs that Python version, creates a virtual environment, and sets up pip. Phase 3: Dependency Installation — The agent installs all necessary dependencies, including build tools like setuptools and wheel, followed by the package itself with its test dependencies (typically using pip install -e ".[test]"). Phase 4: Switching to the Base Commit — After the environment is ready, the agent switches to the base_commit. This is the exact state of the codebase where the bug exists—before any fix was applied. Phase 5: Problem Dispatch to Participant — The green agent sends the problem statement and any hints to the participant agent via the A2A protocol. The participant agent is expected to analyze the problem, explore the codebase using available tools, and return a patch that fixes the issue. Phase 6: Patch Extraction and Application — Once the participant responds, the green agent extracts the patch from the response. It tries multiple application strategies including git apply, git apply --ignore-whitespace, git apply --3way, and patch -p1 with fuzz factors to handle common formatting issues that LLMs may produce. Phase 7: Test Execution and Scoring — The green agent runs two sets of tests. First, it runs the FAIL_TO_PASS tests—these are tests that were failing before the fix and should now pass if the participant's patch correctly addresses the issue. Second, it runs the PASS_TO_PASS tests—these are tests that were already passing before the fix and should continue to pass, ensuring the patch doesn't introduce regressions. Scoring and Final Status Based on the test results, each instance receives one of the following status classifications: Resolved means all FAIL_TO_PASS tests now pass and all PASS_TO_PASS tests continue to pass. This is the ideal outcome indicating a complete and correct fix. Breaking Resolved means the participant successfully fixed all the failing tests, but some previously passing tests now fail. The fix works but introduces regressions elsewhere in the codebase. Partially Resolved means some (but not all) of the failing tests now pass, while all previously passing tests continue to pass. The fix is incomplete but doesn't break anything.

Evaluate an agent’s ability to solve USACO programming problems, including reasoning through complex algorithmic challenges and designing novel solutions under strict time and memory constraints.

Our green evaluator agent benchmarks database-focused agents on Spider2-Snow, a suite of natural-language-to-SQL tasks grounded in Snowflake-backed datasets. For each test instance, it provides the target agent with the instruction, db_id, and any optional external knowledge, and expects a structured response containing a single SQL query (via an A2A DataPart like {"sql": "..."}; plain-text and fenced ```sql fallbacks are also supported). The evaluator then executes the predicted SQL on Snowflake and compares the resulting output to gold execution results to score correctness.