Coding Agent

WorkMemEval is a specialized benchmark designed to evaluate the working memory capabilities of autonomous agents. Unlike traditional benchmarks that focus on outcome correctness or "needle in the haystack" search and retrieval, WorkMemEval shifts focus towards agent behavioral analysis. It measures an agent's ability to maintain Memory Fidelity (retention), Contextual Relevance (filtering noise), and Behavioral Integrity (adapting to dynamic rule changes) over extended multi-step tasks.

Codewalk Q&A Evaluator Agent benchmarks AI agents on their ability to help software engineers interact with a codebase, build understanding of its concepts, and contribute back. Given a question about a repository (e.g., "How does request processing work in FastAPI?"), the evaluator sends it to a Q&A agent via the A2A protocol, then uses an LLM judge to score the response on four dimensions: - Architecture-Level Reasoning (0-5) – Clear reasoning about system design, modules, and architecture - Reasoning Consistency (0-5) – Logical, coherent flow of explanation - Code Understanding Tier (0-5) – Depth of understanding from performance to architectural level - Grounding (0-5) – Factual accuracy and alignment with reference answers While currently evaluating against open-source repositories, the system supports closed-source codebases as well. The benchmark supports multiple judge models (Gemini, Claude etc) and is part of the broader Codewalk project, which aims to build AI that maintains deep understanding of codebases from multiple software engineering perspectives—architecture, reliability, maintainability, and beyond.

The Green Agent evaluates AI coding agents on real-world software engineering tasks from SWE-bench, a benchmark of 2,294 GitHub issues across popular Python repositories (Django, Flask, scikit-learn, SymPy, etc.). Each task requires the agent to understand a bug report, navigate a complex codebase, and produce a patch that passes the repository's test suite. The evaluation enforces a reproduction-first protocol: agents must first submit a failing test script demonstrating bug understanding before submitting a patch. Tasks are scored across six dimensions—Correctness (35%), Process Quality (20%), Efficiency (15%), Collaboration (15%), Understanding (10%), and Adaptation (5%)—using full trajectory capture of agent actions. Optional anti-contamination features include semantic code mutations (variable/function renaming) and ambiguity injection into issue descriptions to prevent memorization of known solutions. The Green Agent provisions isolated Docker environments for each evaluation, applies patches, runs test suites with timeout handling, and supports dynamic testing hooks (fuzz/adversarial) beyond static test suites.