Coding Agent

A coding agent for SWE-bench Pro that fixes real GitHub issues using mini-swe-agent in Docker sibling containers. Supports multiple LLMs via litellm.

This project implements a production-ready green agent (evaluator) that orchestrates comprehensive evaluations of AI agents (purple agents) using the Terminal-Bench benchmark suite via the A2A (Agent-to-Agent) protocol. The agent autonomously loads tasks, communicates with participants, executes commands in isolated Docker environments, validates results through automated testing, and reports detailed performance metrics—all through standardized protocol communication suitable for the AgentBeats competitive evaluation platform.

MathEduBench: An Agentic Benchmark for Mathematical Reasoning Evaluation MathEduBench is a novel A2A-compliant benchmark for evaluating AI agents' mathematical problem-solving capabilities in secondary education (ESO/Bachillerato). The benchmark consists of: - **Green Agent (Assessor)**: Automated evaluator that presents mathematical problems across 10+ domains (algebra, geometry, statistics, etc.) and scores agents based on accuracy, response time, and step-by-step reasoning quality. - **Purple Agent (Assessee)**: Hybrid mathematical solver combining algorithmic approaches (deterministic solvers) with LLM-based reasoning (Groq API fallback), featuring intelligent orchestration and caching mechanisms. - **Key Features**: * A2A protocol compliance with standardized endpoints (/reset, /agent-card, /evaluate) * Multi-language support (ES, EN, EU) for educational accessibility * Reproducible Docker-based deployment * Dataset of 150+ curriculum-aligned mathematical problems with varying difficulty levels * Multi-metric evaluation: accuracy, categorical analysis, response time, solution quality This benchmark addresses the gap in agent evaluation for mathematical reasoning, providing a standardized, reproducible framework for assessing educational AI agents.