Agent Eval
Head-to-head comparison of coding agents (Claude Code, Aider, Codex, etc.) on custom tasks with pass rate, cost, time, and consistency metrics. Use when choosing between coding agents, or when a change to an agent setup needs measured pass rate, cost, and time rather than an impression.
Get the complete skill folder, including its scripts and reference files.
·
When to use this skill
Use Agent Eval when an AI agent needs a reusable SKILL.md workflow for this job: Head-to-head comparison of coding agents (Claude Code, Aider, Codex, etc.) on custom tasks with pass rate, cost, time, and consistency metrics. Use when choosing between coding agents, or when a change to an agent setup needs measured pass rate, cost, and time rather than an impression.
When not to use it
Skip Agent Eval when the task is outside the coding category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.
How to install
Where will you use it?
mkdir -p ~/.claude/skills/agent-eval # Save SKILL.md to ~/.claude/skills/agent-eval/SKILL.md
- Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/agent-eval/SKILL.md. For a shared project, commit the folder at .claude/skills/agent-eval/ instead.
- Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
- For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.
Full install guide for Claude, Cursor, and Codex →
Try a starter prompt
After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.
Use the Agent Eval skill for this task: Use Agent Eval when this documented workflow matches the task. Confirm any required inputs or files before making changes.
What supports this recommendation?
76/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.
- Documentation
- 85/100
- Practical detail
- 76/100
- Source evidence
- 63/100
- Source signals
- 72/100
Model compatibility evidence
- claudeinferred
The skill text mentions Claude or a closely associated term. This is keyword-derived compatibility; no hands-on model test is recorded.
- chatgptinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- geminiinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- copilotinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- llamainferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- perplexityinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- mistralinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- grokinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.
What this skill does
# Agent Eval Skill
A lightweight CLI tool for comparing coding agents head-to-head on reproducible tasks. Every "which coding agent is best?" comparison runs on vibes — this tool systematizes it.
## When to Activate
- Comparing coding agents (Claude Code, Aider, Codex, etc.) on your own codebase - Measuring agent performance before adopting a new tool or model - Running regression checks when an agent updates its model or tooling - Producing data-backed agent selection decisions for a team
## Installation
> **Note:** Install agent-eval from its repository after reviewing the source.
## Core Concepts
### YAML Task Definitions
Define tasks declaratively. Each task specifies what to do, which files to touch, and how to judge success:
```yaml name: add-retry-logic description: Add exponential backoff retry to the HTTP client repo: ./my-project files: - src/http_client.py prompt: | Add retry logic with exponential backoff to all HTTP requests. Max 3 retries. Initial delay 1s, max delay 30s. judge: - type: pytest command: pytest tests/test_http_client.py -v - type: grep pattern: "exponential_backoff|retry" files: src/http_client.py commit: "abc1234" # pin to specific commit for reproducibility ```
### Git Worktree Isolation
Each agent run gets its own git worktree — no Docker required. This provides reproducibility isolation so agents cannot interfere with each other or corrupt the base repo.
### Metrics Collected
| Metric | What It Measures | |--------|-----------------| | Pass rate | Did the agent produce code that passes the judge? | | Cost | API spend per task (when available) | | Time | Wall-clock seconds to completion | | Consistency | Pass rate across repeated runs (e.g., 3/3 = 100%) |
## Workflow
### 1. Define Tasks
Create a `tasks/` directory with YAML files, one per task:
```bash mkdir tasks # Write task definitions (see template above) ```
### 2. Run Agents
Execute agents against your tasks:
```bash agent-eval run --task tasks/add-retry-logic.yaml --agent claude-code --agent aider --runs 3 ```
Each run: 1. Creates a fresh git worktree from the specified commit 2. Hands the prompt to the agent 3. Runs the judge criteria 4. Records pass/fail, cost, and time
### 3. Compare Results
Generate a comparison report:
```bash agent-eval report --format table ```
``` Task: add-retry-logic (3 runs each) ┌──────────────┬───────────┬────────┬────────┬─────────────┐ │ Agent │ Pass Rate │ Cost │ Time │ Consistency │ ├──────────────┼───────────┼────────┼────────┼─────────────┤ │ claude-code │ 3/3 │ $0.12 │ 45s │ 100% │ │ aider │ 2/3 │ $0.08 │ 38s │ 67% │ └──────────────┴───────────┴────────┴────────┴─────────────┘ ```
## Judge Types
### Code-Based (deterministic)
```yaml judge: - type: pytest command: pytest tests/ -v - type: command command: npm run build ```
### Pattern-Based
```yaml judge: - type: grep pattern: "class.*Retry" files: src/**/*.py ```
### Model-Based (LLM-as-judge)
```yaml judge: - type: llm prompt: | Does this implementation correctly handle exponential backoff? Check for: max retries, increasing delays, jitter. ```
## Best Practices
- **Start with 3-5 tasks** that represent your real workload, not toy examples - **Run at least 3 trials** per agent to capture variance — agents are non-deterministic - **Pin the commit** in your task YAML so results are reproducible across days/weeks - **Include at least one deterministic judge** (tests, build) per task — LLM judges add noise - **Track cost alongside pass rate** — a 95% agent at 10x the cost may not be the right choice - **Version your task definitions** — they are test fixtures, treat them as code
## Links
- Repository: [github.com/joaquinhuigomez/agent-eval](https://github.com/joaquinhuigomez/agent-eval)
Intended uses
- Use Agent Eval when this documented workflow matches the task.
Implementation notes
- Start with 3-5 tasks that represent your real workload, not toy examples
- Run at least 3 trials per agent to capture variance — agents are non-deterministic
- Pin the commit in your task YAML so results are reproducible across days/weeks
- Include at least one deterministic judge (tests, build) per task — LLM judges add noise
- Track cost alongside pass rate — a 95% agent at 10x the cost may not be the right choice
- Version your task definitions — they are test fixtures, treat them as code
Try it, then tell us how it went
A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.