Ai Regression Testing
Regression testing strategies for AI-assisted development. Sandbox-mode API testing without database dependencies, automated bug-check workflows, and patterns to catch AI blind spots where the same model writes and reviews code. Use when adding regression coverage to AI-assisted code, or when the same model both wrote and reviewed a change.
Get the complete skill folder, including its scripts and reference files.
·
When to use this skill
Use Ai Regression Testing when an AI agent needs a reusable SKILL.md workflow for this job: Regression testing strategies for AI-assisted development. Sandbox-mode API testing without database dependencies, automated bug-check workflows, and patterns to catch AI blind spots where the same model writes and reviews code. Use when adding regression coverage to AI-assisted code, or when the same model both wrote and reviewed a change.
When not to use it
Skip Ai Regression Testing when the task is outside the coding category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.
How to install
Where will you use it?
mkdir -p ~/.claude/skills/ai-regression-testing # Save SKILL.md to ~/.claude/skills/ai-regression-testing/SKILL.md
- Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/ai-regression-testing/SKILL.md. For a shared project, commit the folder at .claude/skills/ai-regression-testing/ instead.
- Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
- For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.
Full install guide for Claude, Cursor, and Codex →
Try a starter prompt
After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.
Use the Ai Regression Testing skill for this task: Use Ai Regression Testing when this documented workflow matches the task. Confirm any required inputs or files before making changes.
What supports this recommendation?
71/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.
- Documentation
- 77/100
- Practical detail
- 68/100
- Source evidence
- 63/100
- Source signals
- 72/100
Model compatibility evidence
- claudeinferred
The skill text mentions Claude or a closely associated term. This is keyword-derived compatibility; no hands-on model test is recorded.
- chatgptuntested
No model-specific signal or recorded compatibility test was found.
- geminiuntested
No model-specific signal or recorded compatibility test was found.
- copilotuntested
No model-specific signal or recorded compatibility test was found.
- llamauntested
No model-specific signal or recorded compatibility test was found.
- perplexityuntested
No model-specific signal or recorded compatibility test was found.
- mistraluntested
No model-specific signal or recorded compatibility test was found.
- grokuntested
No model-specific signal or recorded compatibility test was found.
Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.
What this skill does
# AI Regression Testing
Testing patterns specifically designed for AI-assisted development, where the same model writes code and reviews it — creating systematic blind spots that only automated tests can catch.
## When to Activate
- AI agent (Claude Code, Cursor, Codex) has modified API routes or backend logic - A bug was found and fixed — need to prevent re-introduction - Project has a sandbox/mock mode that can be leveraged for DB-free testing - Running `/bug-check` or similar review commands after code changes - Multiple code paths exist (sandbox vs production, feature flags, etc.)
## The Core Problem
When an AI writes code and then reviews its own work, it carries the same assumptions into both steps. This creates a predictable failure pattern:
``` AI writes fix → AI reviews fix → AI says "looks correct" → Bug still exists ```
**Real-world example** (observed in production):
``` Fix 1: Added notification_settings to API response → Forgot to add it to the SELECT query → AI reviewed and missed it (same blind spot)
Fix 2: Added it to SELECT query → TypeScript build error (column not in generated types) → AI reviewed Fix 1 but didn't catch the SELECT issue
Fix 3: Changed to SELECT * → Fixed production path, forgot sandbox path → AI reviewed and missed it AGAIN (4th occurrence)
Fix 4: Test caught it instantly on first run PASS: ```
The pattern: **sandbox/production path inconsistency** is the #1 AI-introduced regression.
## Sandbox-Mode API Testing
Most projects with AI-friendly architecture have a sandbox/mock mode. This is the key to fast, DB-free API testing.
### Setup (Vitest + Next.js App Router)
```typescript // vitest.config.ts import { defineConfig } from "vitest/config"; import path from "path";
export default defineConfig({ test: { environment: "node", globals: true, include: ["__tests__/**/*.test.ts"], setupFiles: ["__tests__/setup.ts"], }, resolve: { alias: { "@": path.resolve(__dirname, "."), }, }, }); ```
```typescript // __tests__/setup.ts // Force sandbox mode — no database needed process.env.SANDBOX_MODE = "true"; process.env.NEXT_PUBLIC_SUPABASE_URL = ""; process.env.NEXT_PUBLIC_SUPABASE_ANON_KEY = ""; ```
### Test Helper for Next.js API Routes
```typescript // __tests__/helpers.ts import { NextRequest } from "next/server";
export function createTestRequest( url: string, options?: { method?: string; body?: Record<string, unknown>; headers?: Record<string, string>; sandboxUserId?: string; }, ): NextRequest { const { method = "GET", body, headers = {}, sandboxUserId } = options || {}; const fullUrl = url.startsWith("http") ? url : `http://localhost:3000${url}`; const reqHeaders: Record<string, string> = { ...headers };
if (sandboxUserId) { reqHeaders["x-sandbox-user-id"] = sandboxUserId; }
const init: { method: string; headers: Record<string, string>; body?: string } = { method, headers: reqHeaders, };
if (body) { init.body = JSON.stringify(body); reqHeaders["content-type"] = "application/json"; }
return new NextRequest(fullUrl, init); }
export async function parseResponse(response: Response) { const json = await response.json(); return { status: response.status, json }; } ```
### Writing Regression Tests
The key principle: **write tests for bugs that were found, not for code that works**.
```typescript // __tests__/api/user/profile.test.ts import { describe, it, expect } from "vitest"; import { createTestRequest, parseResponse } from "../../helpers"; import { GET, PATCH } from "@/app/api/user/profile/route";
// Define the contract — what fields MUST be in the response const REQUIRED_FIELDS = [ "id", "email", "full_name", "phone", "role", "created_at", "avatar_url", "notification_settings", // ← Added after bug found it missing ];
describe("GET /api/user/profile", () => { it("returns all required fields", async () => { const req = createTestRequest("/api/user/profile"); const res = await GET(req); const { status, json } = await parseResponse(res);
expect(status).toBe(200); for (const field of REQUIRED_FIELDS) { expect(json.data).toHaveProperty(field); } });
// Regression test — this exact bug was introduced by AI 4 times it("notification_settings is not undefined (BUG-R1 regression)", async () => { const req = createTestRequest("/api/user/profile"); const res = await GET(req); const { json } = await parseResponse(res);
expect("notification_settings" in json.data).toBe(true); const ns = json.data.notification_settings; expect(ns === null || typeof ns === "object").toBe(true); }); }); ```
### Testing Sandbox/Production Parity
The most common AI regression: fixing production path but forgetting sandbox path (or vice versa).
```typescript // Test that sandbox responses match the expected contract describe("GET /api/user/messages (conversation list)", () => { it("includes partner_name in sandbox mode", async () => { const req = createTestRequest("/api/user/messages", { sandboxUserId: "user-001", }); const res = await GET(req); const { json } = await parseResponse(res);
// This caught a bug where partner_name was added // to production path but not sandbox path if (json.data.length > 0) { for (const conv of json.data) { expect("partner_name" in conv).toBe(true); } } }); }); ```
## Integrating Tests into Bug-Check Workflow
### Custom Command Definition
```markdown <!-- .claude/commands/bug-check.md --> # Bug Check
## Step 1: Automated Tests (mandatory, cannot skip)
Run these commands FIRST before any code review:
npm run test # Vitest test suite npm run build # TypeScript type check + build
- If tests fail → report as highest priority bug - If build fails → report type errors as highest priority - Only proceed to Step 2 if both pass
## Step 2: Code Review (AI review)
1. Sandbox / production path consistency 2. API response shape matches frontend expectations 3. SELECT clause completeness 4. Error handling with rollback 5. Optimistic update race conditions
## Step 3: For each bug fixed, propose a regression test ```
### The Workflow
``` User: "バグチェックして" (or "/bug-check") │ ├─ Step 1: npm run test │ ├─ FAIL → Bug found mechanically (no AI judgment needed) │ └─ PASS → Continue │ ├─ Step 2: npm run build │ ├─ FAIL → Type error found mechanically │ └─ PASS → Continue │ ├─ Step 3: AI code review (with known blind spots in mind) │ └─ Findings reported │ └─ Step 4: For each fix, write a regression test └─ Next bug-check catches if fix breaks ```
## Common AI Regression Patterns
### Pattern 1: Sandbox/Production Path Mismatch
**Frequency**: Most common (observed in 3 out of 4 regressions)
```typescript // FAIL: AI adds field to production path only if (isSandboxMode()) { return { data: { id, email, name } }; // Missing new field } // Production path return { data: { id, email, name, notification_settings } };
// PASS: Both paths must return the same shape if (isSandboxMode()) { return { data: { id, email, name, notification_settings: null } }; } return { data: { id, email, name, notification_settings } }; ```
**Test to catch it**:
```typescript it("sandbox and production return same fields", async () => { // In test env, sandbox mode is forced ON const res = await GET(createTestRequest("/api/user/profile")); const { json } = await parseResponse(res);
for (const field of REQUIRED_FIELDS) { expect(json.data).toHaveProperty(field); } }); ```
### Pattern 2: SELECT Clause Omission
**Frequency**: Common with Supabase/Prisma when adding new columns
```typescript // FAIL: New column added to response but not to SELECT const { data } = await supabase .from("users") .select("id, email, name") // notification_settings not here .single();
return { data: { ...data, notification_settings: data.notification_settings } }; // → notification_settings is always undefined
// PASS: Use SELECT * or explicitly include new columns const { data } = await supabase .from("users") .select("*") .single(); ```
### Pattern 3: Error State Leakage
**Frequency**: Moderate — when adding error handling to existing components
```typescript // FAIL: Error state set but old data not cleared catch (err) { setError("Failed to load"); // reservations still shows data from previous tab! }
// PASS: Clear related state on error catch (err) { setReservations([]); // Clear stale data setError("Failed to load"); } ```
### Pattern 4: Optimistic Update Without Proper Rollback
```typescript // FAIL: No rollback on failure const handleRemove = async (id: string) => { setItems(prev => prev.filter(i => i.id !== id)); await fetch(`/api/items/${id}`, { method: "DELETE" }); // If API fails, item is gone from UI but still in DB };
// PASS: Capture previous state and rollback on failure const handleRemove = async (id: string) => { const prevItems = [...items]; setItems(prev => prev.filter(i => i.id !== id)); try { const res = await fetch(`/api/items/${id}`, { method: "DELETE" }); if (!res.ok) throw new Error("API error"); } catch { setItems(prevItems); // Rollback alert("削除に失敗しました"); } }; ```
## Strategy: Test Where Bugs Were Found
Don't aim for 100% coverage. Instead:
``` Bug found in /api/user/profile → Write test for profile API Bug found in /api/user/messages → Write test for messages API Bug found in /api/user/favorites → Write test for favorites API No bug in /api/user/notifications → Don't write test (yet) ```
**Why this works with AI development:**
1. AI tends to make the **same category of mistake** repeatedly 2. Bugs cluster in complex areas (auth, multi-path logic, state management) 3. Once tested, that exact regression **cannot happen again** 4. Test count grows organically with bug fixes — no wasted effort
## Quick Reference
| AI Regression Pattern | Test Strategy | Priority | |---|---|---| | Sandbox/production mismatch | Assert same response shape in sandbox mode | High | | SELECT clause omission | Assert all required fields in response | High | | Error state leakage | Assert state cleanup on error | Medium | | Missing rollback | Assert state restored on API failure | Medium | | Type cast masking null | Assert field is not undefined | Medium |
## DO / DON'T
**DO:** - Write tests immediately after finding a bug (before fixing it if possible) - Test the API response shape, not the implementation - Run tests as the first step of every bug-check - Keep tests fast (< 1 second total with sandbox mode) - Name tests after the bug they prevent (e.g., "BUG-R1 regression")
**DON'T:** - Write tests for code that has never had a bug - Trust AI self-review as a substitute for automated tests - Skip sandbox path testing because "it's just mock data" - Write integration tests when unit tests suffice - Aim for coverage percentage — aim for regression prevention
Intended uses
- Use Ai Regression Testing when this documented workflow matches the task.
Implementation notes
- Review the source instructions and adapt inputs before running the workflow.
Try it, then tell us how it went
A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.