Skip to main content
Agent skill

Eval Harness

Eval-driven development (EDD) framework for AI coding sessions — define capability and regression evals before coding, grade with code-based, model-based, rule, or human graders, and track pass@k and pass^k reliability. Use when defining pass/fail criteria for agent tasks, measuring agent reliability, building regression suites for prompt or agent changes, or benchmarking across model versions.

Get started

Get the complete skill folder, including its scripts and reference files.

·

When to use this skill

Use Eval Harness when an AI agent needs a reusable SKILL.md workflow for this job: Eval-driven development (EDD) framework for AI coding sessions — define capability and regression evals before coding, grade with code-based, model-based, rule, or human graders, and track pass@k and pass^k reliability. Use when defining pass/fail criteria for agent tasks, measuring agent reliability, building regression suites for prompt or agent changes, or benchmarking across model versions.

When not to use it

Skip Eval Harness when the task is outside the coding category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.

How to install

Where will you use it?

Install scope
Claude Code folder setup
mkdir -p ~/.claude/skills/eval-harness
# Save SKILL.md to ~/.claude/skills/eval-harness/SKILL.md
  1. Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/eval-harness/SKILL.md. For a shared project, commit the folder at .claude/skills/eval-harness/ instead.
  2. Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
  3. For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.

Full install guide for Claude, Cursor, and Codex →

Try a starter prompt

After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.

Starter prompt
Use the Eval Harness skill for this task: Use Eval Harness when this documented workflow matches the task. Confirm any required inputs or files before making changes.

What supports this recommendation?

76/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.

Documentation
85/100
Practical detail
76/100
Source evidence
63/100
Source signals
72/100
Model compatibility evidence
  • claudeinferred

    The skill text mentions Claude or a closely associated term. This is keyword-derived compatibility; no hands-on model test is recorded.

  • chatgptinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • geminiinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • copilotinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • llamainferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • perplexityinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • mistralinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • grokinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.

What this skill does

# Eval Harness Skill

A formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles.

## When to Activate

- Setting up eval-driven development (EDD) for AI-assisted workflows - Defining pass/fail criteria for Claude Code task completion - Measuring agent reliability with pass@k metrics - Creating regression test suites for prompt or agent changes - Benchmarking agent performance across model versions

## Philosophy

Eval-Driven Development treats evals as the "unit tests of AI development": - Define expected behavior BEFORE implementation - Run evals continuously during development - Track regressions with each change - Use pass@k metrics for reliability measurement

## Eval Types

### Capability Evals Test if Claude can do something it couldn't before: ```markdown [CAPABILITY EVAL: feature-name] Task: Description of what Claude should accomplish Success Criteria: - [ ] Criterion 1 - [ ] Criterion 2 - [ ] Criterion 3 Expected Output: Description of expected result ```

### Regression Evals Ensure changes don't break existing functionality: ```markdown [REGRESSION EVAL: feature-name] Baseline: SHA or checkpoint name Tests: - existing-test-1: PASS/FAIL - existing-test-2: PASS/FAIL - existing-test-3: PASS/FAIL Result: X/Y passed (previously Y/Y) ```

## Grader Types

### 1. Code-Based Grader Deterministic checks using code: ```bash # Check if file contains expected pattern grep -q "export function handleAuth" src/auth.ts && echo "PASS" || echo "FAIL"

# Check if tests pass npm test -- --testPathPattern="auth" && echo "PASS" || echo "FAIL"

# Check if build succeeds npm run build && echo "PASS" || echo "FAIL" ```

### 2. Model-Based Grader Use Claude to evaluate open-ended outputs: ```markdown [MODEL GRADER PROMPT] Evaluate the following code change: 1. Does it solve the stated problem? 2. Is it well-structured? 3. Are edge cases handled? 4. Is error handling appropriate?

Score: 1-5 (1=poor, 5=excellent) Reasoning: [explanation] ```

### 3. Human Grader Flag for manual review: ```markdown [HUMAN REVIEW REQUIRED] Change: Description of what changed Reason: Why human review is needed Risk Level: LOW/MEDIUM/HIGH ```

## Metrics

### pass@k "At least one success in k attempts" - pass@1: First attempt success rate - pass@3: Success within 3 attempts - Typical target: pass@3 > 90%

### pass^k "All k trials succeed" - Higher bar for reliability - pass^3: 3 consecutive successes - Use for critical paths

## Eval Workflow

### 1. Define (Before Coding) ```markdown ## EVAL DEFINITION: feature-xyz

### Capability Evals 1. Can create new user account 2. Can validate email format 3. Can hash password securely

### Regression Evals 1. Existing login still works 2. Session management unchanged 3. Logout flow intact

### Success Metrics - pass@3 > 90% for capability evals - pass^3 = 100% for regression evals ```

### 2. Implement Write code to pass the defined evals.

### 3. Evaluate ```bash # Run capability evals [Run each capability eval, record PASS/FAIL]

# Run regression evals npm test -- --testPathPattern="existing"

# Generate report ```

### 4. Report ```markdown EVAL REPORT: feature-xyz ========================

Capability Evals: create-user: PASS (pass@1) validate-email: PASS (pass@2) hash-password: PASS (pass@1) Overall: 3/3 passed

Regression Evals: login-flow: PASS session-mgmt: PASS logout-flow: PASS Overall: 3/3 passed

Metrics: pass@1: 67% (2/3) pass@3: 100% (3/3)

Status: READY FOR REVIEW ```

## Integration Patterns

### Pre-Implementation ``` /eval define feature-name ``` Creates eval definition file at `.claude/evals/feature-name.md`

### During Implementation ``` /eval check feature-name ``` Runs current evals and reports status

### Post-Implementation ``` /eval report feature-name ``` Generates full eval report

## Eval Storage

Store evals in project: ``` .claude/ evals/ feature-xyz.md # Eval definition feature-xyz.log # Eval run history baseline.json # Regression baselines ```

## Best Practices

1. **Define evals BEFORE coding** - Forces clear thinking about success criteria 2. **Run evals frequently** - Catch regressions early 3. **Track pass@k over time** - Monitor reliability trends 4. **Use code graders when possible** - Deterministic > probabilistic 5. **Human review for security** - Never fully automate security checks 6. **Keep evals fast** - Slow evals don't get run 7. **Version evals with code** - Evals are first-class artifacts

## Example: Adding Authentication

```markdown ## EVAL: add-authentication

### Phase 1: Define (10 min) Capability Evals: - [ ] User can register with email/password - [ ] User can login with valid credentials - [ ] Invalid credentials rejected with proper error - [ ] Sessions persist across page reloads - [ ] Logout clears session

Regression Evals: - [ ] Public routes still accessible - [ ] API responses unchanged - [ ] Database schema compatible

### Phase 2: Implement (varies) [Write code]

### Phase 3: Evaluate Run: /eval check add-authentication

### Phase 4: Report EVAL REPORT: add-authentication ============================== Capability: 5/5 passed (pass@3: 100%) Regression: 3/3 passed (pass^3: 100%) Status: SHIP IT ```

## Local Framework Utilities

The mechanical utilities ship in `scripts/lib/eval-harness/`:

```sh node scripts/eval-harness.js example ```

- Capsule: hash-linked journal with five lineages and local integrity checks. - Inspection: source digests, validated variant paths, and syntactic warnings. - Replay: declared tools and content-addressed fixtures. Missing fixtures fail closed; SE3 and above are refused in replay. Record mode invokes the registered implementation, so only register trusted functions. - Receipt: offline verification of capsule and artifact bytes, with named checks. - Retrospective preparation: `node scripts/eval-harness.js capsule group <dir> [<dir> ...]` groups 1 to 100 explicitly selected, verified local capsule snapshots from one task family by declared harness version. Repeated snapshots count once; conflicting identities or invalid capsules reject the whole report. This is read-only record counting, with no new rollouts, scores or promotion. Use small, quiescent capsules. Payloads, directory arguments and raw run/capsule IDs are omitted, but task-family/version labels are verbatim and digest references are linkable; review them before sharing. Operational validation remains pending.

Candidate execution is disabled on every OS because no verified OS containment backend is implemented. `gate run`, `runGate`, `runVariant`, direct child launch, and the retired effect preload refuse with `gate.isolation_required`. No trust flag or caller-supplied executor can bypass the refusal. The example records that refusal and inspects source without executing or scoring it.

Do not present static warnings, a capsule receipt, or successful utility tests as candidate containment or promotion evidence. A future gate requires an independently reviewed OS boundary, protected checker and audit channels, and fatal baseline rejection. See `docs/architecture/eval-harness-frameworks.md`.

## Product Evals (v1.8)

Use product evals when behavior quality cannot be captured by unit tests alone.

### Grader Types

1. Code grader (deterministic assertions) 2. Rule grader (regex/schema constraints) 3. Model grader (LLM-as-judge rubric) 4. Human grader (manual adjudication for ambiguous outputs)

### pass@k Guidance

- `pass@1`: direct reliability - `pass@3`: practical reliability under controlled retries - `pass^3`: stability test (all 3 runs must pass)

Recommended thresholds: - Capability evals: pass@3 >= 0.90 - Regression evals: pass^3 = 1.00 for release-critical paths

### Eval Anti-Patterns

- Overfitting prompts to known eval examples - Measuring only happy-path outputs - Ignoring cost and latency drift while chasing pass rates - Allowing flaky graders in release gates

### Minimal Eval Artifact Layout

- `.claude/evals/<feature>.md` definition - `.claude/evals/<feature>.log` run history - `docs/releases/<version>/eval-summary.md` release snapshot

Intended uses

  • Use Eval Harness when this documented workflow matches the task.

Implementation notes

  • Define evals BEFORE coding - Forces clear thinking about success criteria
  • Run evals frequently - Catch regressions early
  • Track pass@k over time - Monitor reliability trends
  • Use code graders when possible - Deterministic > probabilistic
  • Human review for security - Never fully automate security checks
  • Keep evals fast - Slow evals don't get run
  • Version evals with code - Evals are first-class artifacts

Try it, then tell us how it went

A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.