Skip to main content
Agent skill

Handle Regression

Protocol for detecting, isolating, and fixing regressions in test262 pass count. Use when a test run shows fewer passes than expected.

Get started

Get the complete skill folder, including its scripts and reference files.

When to use this skill

Use Handle Regression when an AI agent needs a reusable SKILL.md workflow for this job: Protocol for detecting, isolating, and fixing regressions in test262 pass count. Use when a test run shows fewer passes than expected.

When not to use it

Skip Handle Regression when the task is outside the coding category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.

How to install

Where will you use it?

Install scope
Claude Code folder setup
mkdir -p ~/.claude/skills/handle-regression
# Save SKILL.md to ~/.claude/skills/handle-regression/SKILL.md
  1. Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/handle-regression/SKILL.md. For a shared project, commit the folder at .claude/skills/handle-regression/ instead.
  2. Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
  3. For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.

Full install guide for Claude, Cursor, and Codex →

Try a starter prompt

After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.

Starter prompt
Use the Handle Regression skill for this task: Use Handle Regression when this documented workflow matches the task. Confirm any required inputs or files before making changes.

What supports this recommendation?

69/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.

Documentation
74/100
Practical detail
68/100
Source evidence
58/100
Source signals
72/100
Model compatibility evidence
  • claudeinferred

    The skill text mentions Claude or a closely associated term. This is keyword-derived compatibility; no hands-on model test is recorded.

  • chatgptuntested

    No model-specific signal or recorded compatibility test was found.

  • geminiuntested

    No model-specific signal or recorded compatibility test was found.

  • copilotuntested

    No model-specific signal or recorded compatibility test was found.

  • llamauntested

    No model-specific signal or recorded compatibility test was found.

  • perplexityuntested

    No model-specific signal or recorded compatibility test was found.

  • mistraluntested

    No model-specific signal or recorded compatibility test was found.

  • grokuntested

    No model-specific signal or recorded compatibility test was found.

Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.

What this skill does

# Handle Regression Protocol

Use this when a test262 run shows fewer passes than the baseline or previous run.

## Step 1: Confirm the regression is real

```bash # Compare current vs archived results python3 -c " import json, os current = {} with open('benchmarks/results/test262-results.jsonl') as f: for line in f: r = json.loads(line) current[r['file']] = r

archives = sorted([f for f in os.listdir('benchmarks/results') if f.startswith('test262-results-') and f.endswith('.jsonl')]) if not archives: print('No archived results to compare. Run was the first.') exit()

prev = {} with open(f'benchmarks/results/{archives[-1]}') as f: for line in f: r = json.loads(line) prev[r['file']] = r

regressions = [(t, prev[t]['status'], current[t]['status'], current[t].get('error','')[:60]) for t in prev if prev[t]['status'] == 'pass' and current.get(t,{}).get('status') != 'pass'] fixes = [t for t in current if current[t]['status'] == 'pass' and prev.get(t,{}).get('status') != 'pass']

print(f'Regressions: {len(regressions)}, Fixes: {len(fixes)}, Net: {len(fixes) - len(regressions):+d}')

from collections import Counter patterns = Counter() for t, old, new, err in regressions: if 'SyntaxError' in err: patterns['expected SyntaxError but compiled'] += 1 elif 'null' in err.lower(): patterns['null deref'] += 1 elif new == 'compile_error': patterns['new compile error'] += 1 elif 'returned' in err: patterns['wrong return value'] += 1 else: patterns[err[:30] or new] += 1

print() print('Regression patterns:') for p, c in patterns.most_common(10): print(f' {c:4d} {p}')

print() print('Sample regressions (1 per pattern):') shown = set() for t, old, new, err in sorted(regressions): for p in patterns: if p not in shown and p[:15] in (err + new)[:60]: print(f' {t}') shown.add(p) break if len(shown) >= 5: break " ```

If regressions < 10 and could be flaky tests, re-run test262 to confirm.

## Step 2: Identify which merge caused it

### Option A: git bisect (fast, automated)

```bash GOOD="<last-known-good-commit>" # from runs/index.json BAD="HEAD"

# Pick one sample test per regression pattern git bisect start git bisect bad $BAD git bisect good $GOOD

# Use vitest filter for a specific regressed test git bisect run npx vitest run tests/test262-vitest.test.ts -t "<test-name>" # Result: "<hash> is the first bad commit" git bisect reset ```

Note: git bisect can be confused by merge commits. Use `--first-parent` if results seem wrong, or bisect each regression pattern separately.

### Option B: test at merge boundaries (manual but reliable)

If bisect is unreliable (merge commits), test at each merge point:

```bash # List code merges between good and bad git log --oneline --first-parent $GOOD..HEAD | grep -E "feat|fix|Merge"

# For each suspect merge, checkout and test git stash git checkout <merge-commit> npx vitest run tests/test262-vitest.test.ts -t "<test-name>" 2>&1 | grep -E "passed|failed" git checkout main git stash pop ```

### Option C: test on dev worktrees (if still available)

```bash # Each dev's worktree has their code in isolation git worktree list

# Test the sample on each worktree for wt in /workspace/.claude/worktrees/*/; do echo "=== $(basename $wt) ===" cd "$wt" && npx vitest run tests/test262-vitest.test.ts -t "<test>" 2>&1 | grep -E "passed|failed" done cd /workspace ```

## Step 3: Reopen issues and resume sprint

For each regression source:

1. **Update the offending issue frontmatter**: set `status: ready`, add `regression: true` 3. **Add a `## Regression` section** to the issue file (preserve existing `## Implementation Notes`): ```markdown ## Regression (sprint-{N})

**Detected**: YYYY-MM-DD via test262 run **Tests affected**: N regressions (pattern: ...) **Sample test**: test/path/to/regressed-test.js **Root cause**: [what the original fix did wrong] **Fix approach**: [how to fix without losing the original improvement] **Bisect result**: [commit hash] was the first bad commit

## Regression Fix

**Commit**: [hash] **What changed**: [description] **Test results**: X/Y regressed tests restored, original fix preserved: yes/no ```

The issue file becomes the full history: problem → fix → regression → regression fix. 4. **Update the sprint doc** (`plan/issues/sprints/{N}/sprint.md`): mark the issue as regressed in the task table 5. **Update dependency graph**: re-add the issue as ready 6. **Create tasks** for each regression pattern, referencing the reopened issue

The sprint is NOT done until all regressions are fixed. The sprint results must show pass count >= baseline.

## Step 4: Assign and fix

Each regression pattern may need a different fix. Create one task per pattern:

| Pattern | Count | Likely culprit | Task | |---------|-------|---------------|------| | SyntaxError | N | #NNN (try/catch wrapping?) | #task | | null deref | N | #NNN (cast guards?) | #task | | new CE | N | #NNN (repair pass?) | #task |

Assign each to a dev. They work in parallel on separate files.

**No dev is released from a regression task without running full test262 and confirming pass count recovery.** "It looks fixed" or "sampled tests pass" is not sufficient — the full suite must run.

## Step 5: Fix and verify

Each dev: 1. Reads the original commit (`git show <hash>`) 2. Understands what the change did and why the regression happens 3. Fixes on their branch — the fix must: - Restore the regressed tests to PASS - AND preserve the original fix's improvements (don't just revert) - If both aren't possible: revert the original and file a new issue for a better approach 4. **Runs full test262 on their integrated branch** (mandatory for regression fixes) 5. Verifies: the specific regressed tests now pass AND the original fix still works 6. Creates test proof, merges via ff-only

## Step 6: Verify full recovery

After all regression fixes are merged:

```bash pnpm run test:262 ```

Compare pass count to the baseline. Must be **>= baseline**. Ideally baseline + genuine fixes.

If pass count is still below baseline after all fixes: - Investigate remaining regressions (repeat from Step 1) - If the fix is too complex, revert the original issue and file a new issue with the regression noted - **The sprint does not close with a net regression**

## Step 7: Update sprint doc

```markdown ## Regression incident

**Detected**: [date] **Cause**: [which issues] **Impact**: -N tests **Resolution**: [what was fixed/reverted] **Final result**: [pass count after fix, delta from baseline] **Time spent**: [hours on regression vs productive work] ```

Move fixed issues back to `done/` with the regression notes preserved.

## Prevention (for future sprints)

- **Pre-merge hook requires test proof** — all tests on branch before merging to main - **Full test262 mandatory for core codegen changes** — not just equiv tests - **Archive results before every run** — enables diff analysis - **One merge at a time, test between merges** — catches regressions early when they're small - **Never stack untested merges** — the sprint-31 regression came from 4 merges without testing between them

## Red flags that indicate a regression source

| Red flag | Likely cause | |----------|-------------| | "expected SyntaxError but compiled" | try/catch wrapping catching compile-time errors | | null deref spike | ref.cast guards silently producing null | | new CEs in previously-passing tests | repair pass changing correct code | | tests pass in worktree but fail on main | merge interaction between branches |

Intended uses

  • Use Handle Regression when this documented workflow matches the task.

Implementation notes

  • Review the source instructions and adapt inputs before running the workflow.

Try it, then tell us how it went

A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.