Skip to main content
Agent skill

Simplicio Autoresearch

Evolutionary optimize-by-metric loop — mutate a target, evaluate against fixed criteria, KEEP if the score improves (commit) or REVERT if it doesn't (git checkout), repeat, plateau-break after N stagnated runs. Adapts Karpathy's autoresearch pattern (and the ECC autoresearch-agent) with mandatory yool guardrails (§11 caps), git-isolated branch discipline, an anti-Goodhart eval order (correctness gate FIRST, score SECOND, labeled tokenizer), a local-first mutation ladder, and a simplicio.savings-event/v1 receipt per run. Use when the user says "optimize this by <metric>", "autoresearch loop", "evolutionary optimize", "mutate/eval/keep-revert", or asks to shrink tokens/latency/bundle size/improve pass-rate against a fixed, measurable eval. Worker: scripts/autoresearch.py.

Get started

Get the complete skill folder, including its scripts and reference files.

When to use this skill

Use Simplicio Autoresearch when an AI agent needs a reusable SKILL.md workflow for this job: Evolutionary optimize-by-metric loop — mutate a target, evaluate against fixed criteria, KEEP if the score improves (commit) or REVERT if it doesn't (git checkout), repeat, plateau-break after N stagnated runs. Adapts Karpathy's autoresearch pattern (and the ECC autoresearch-agent) with mandatory yool guardrails (§11 caps), git-isolated branch discipline, an anti-Goodhart eval order (correctness gate FIRST, score SECOND, labeled tokenizer), a local-first mutation ladder, and a simplicio.savings-event/v1 receipt per run. Use when the user says "optimize this by <metric>", "autoresearch loop", "evolutionary optimize", "mutate/eval/keep-revert", or asks to shrink tokens/latency/bundle size/improve pass-rate against a fixed, measurable eval. Worker: scripts/autoresearch.py.

When not to use it

Skip Simplicio Autoresearch when the task is outside the coding category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.

How to install

Where will you use it?

Install scope
Claude Code folder setup
mkdir -p ~/.claude/skills/simplicio-autoresearch
# Save SKILL.md to ~/.claude/skills/simplicio-autoresearch/SKILL.md
  1. Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/simplicio-autoresearch/SKILL.md. For a shared project, commit the folder at .claude/skills/simplicio-autoresearch/ instead.
  2. Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
  3. For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.

Full install guide for Claude, Cursor, and Codex →

Try a starter prompt

After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.

Starter prompt
Use the Simplicio Autoresearch skill for this task: Use Simplicio Autoresearch when this documented workflow matches the task. Confirm any required inputs or files before making changes.

What supports this recommendation?

69/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.

Documentation
77/100
Practical detail
68/100
Source evidence
52/100
Source signals
72/100
Model compatibility evidence
  • claudeinferred

    The skill text mentions Claude or a closely associated term. This is keyword-derived compatibility; no hands-on model test is recorded.

  • chatgptinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • geminiinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • copilotinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • llamainferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • perplexityinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • mistralinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • grokinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.

What this skill does

# simplicio-autoresearch — evolutionary optimize-by-metric loop

Credit: Andrej Karpathy's `autoresearch` pattern (https://github.com/balukosuri/Andrej-Karpathy-s-Autoresearch-As-a-Universal-Skill) and the ECC bundle's `autoresearch-agent`, adapted into a first-class simplicio skill. The upstream loop is **NOT installed raw** — it conflicts with this ecosystem's rules (unbounded iteration, no git isolation discipline, single-metric hill-climbing). This skill fixes all three. The mechanical bookkeeping (caps, git actions, journal, plateau math, receipt) is a deterministic, model-free worker (`scripts/autoresearch.py`) — it never mutates the target itself. **You (the LLM driving this skill) propose the mutation; the worker enforces the contract around it.**

## yool guardrails (§11 — MANDATORY, not optional)

Register this skill's runtime as an agent with a hard cap on iterations and budget BEFORE the first mutation — an uncapped loop is a review-blocker, per spec:

```markdown ### simplicio-autoresearch

- yool_id: `agent.dev.autoresearch` - authority: dev - lane: background - agent_terms: cpu_quota_pct: 60 disk_quota_mb: 100 timeout_s: 300 max_iterations: <N> # MANDATORY — set at `init`, never omitted max_token_budget: <N> # MANDATORY — set at `init`, never omitted ```

`scripts/autoresearch.py init` **refuses to start** (exit 2) without both `--max-iterations` and `--max-token-budget` as positive integers — the cap is enforced mechanically, not by convention. `record` refuses (exit 12) any iteration number beyond the frozen `max_iterations`.

## When to use / when NOT to use

Fit requires: a concrete target file (or small set of files), a command that can score it, and a correctness gate that is genuinely binary.

| Good fit (pilots) | Why | |---|---| | mapper — TOON encoder heuristics | eval = round-trip fixtures (correctness) + measured tokens on real artifacts (score) | | dev-cli — prompt template | eval = the A/B bench pass-rate, fixed validation set | | runtime — local model-ladder tuning | eval = ladder success-rate / latency on a fixed task set |

| Low fit — do NOT use | Why | |---|---| | sprint (planning code vs its own pytest suite) | the eval IS the test suite the code is graded against — the loop would learn to game its own gate (Goodhart) | | an agent already in production | "cache is sacred" — mutating a live, cached agent config outside an offline branch risks correctness regressions no one is watching for | | marketing copy with no fixed judge | a wobbling human/LLM judge with no binary criteria is not a gate, it's noise — the loop would hill-climb on judge mood, not quality |

## Contract — inputs you must have BEFORE calling `init`

- **Target**: one file (or a tight set) that the mutation touches. - **Eval command**: prints `{"gate": "pass"|"fail", "score": <number>}` (JSON anywhere in stdout/ stderr, or `gate: pass` / `score: 7` lines as a fallback) — see `parse_eval_output` in the worker. The eval **includes the repo's OWN gate** (lint + tests for the target's language/repo) — the loop must never commit red. - **Binary correctness criteria FIRST** — round-trip lossless / fixtures / the target repo's test suite. This is the gate; it decides keep vs revert **before** the score is even consulted. - **A labeled metric SECOND** — the score being optimized (tokens via a NAMED tokenizer, pass- rate, latency, bundle size). An unlabeled/unrotulado metric is not a valid score — 5 divergent token estimators disagree by ~30%; hill-climbing on an unlabeled number is a random walk. Pass `--tokenizer-id <name>` at `init` whenever the score is a token count. - **Fixed validation set** (3-5 items), unchanged for the whole run — mutating criteria mid-run invalidates every prior keep/revert decision. - **Caps**: `--max-iterations`, `--max-token-budget` (mandatory), optionally `--cpu-quota-pct` (default 60), `--disk-quota-mb` (default 100), `--timeout-s` (default 300), `--plateau-n` (default 5).

## Step 1 — init (git isolation, baseline)

```bash python3 scripts/autoresearch.py init --target <path> --eval "<eval command>" \ --max-iterations 20 --max-token-budget 50000 [--tokenizer-id cl100k-estimate] \ [--branch <name>] [--direction max|min] [--margin 0.0] [--plateau-n 5] ```

Refuses to start on `main`/`master` — it resolves the target's git root, checks out (creating if needed) an isolated `autoresearch/<slug>` branch, records the pre-mutation `HEAD`, and runs the eval command ONCE to capture the baseline gate + score. Print the `store=` path it returns; every later command needs `--store <that path>` (or `export SIMPLICIO_AUTORESEARCH_STORE=<that path>`).

## Step 2 — the mutate/eval/keep-revert loop (repeat until cap or promise)

For each iteration N (1..max_iterations):

1. **Mutate** — apply ONE change to the target. Prefer the **simplicio-runtime local ladder** (qwen 64→600) for the mutation itself; a paid remote model is stage 5 ONLY, opt-in via `--remote`/`--allow-remote` with recorded escalation evidence. An optimizer that burns frontier tokens indefinitely contradicts the point of the exercise. 2. **Record** — let the worker run the eval and decide: ```bash python3 scripts/autoresearch.py record --iteration <N> --store <store> \ --note "<what you changed>" --mutation-summary "<one-line>" ``` Gate-first, always: a `fail` gate is **always** `revert`, no matter how good the score looks (anti-Goodhart). Only once the gate passes does the score decide keep vs revert, by `--direction` (`max` default; `min` for tokens/latency/size) beyond `--margin`. `record` performs the matching **scoped** git action itself — `git add <target> && git commit` on keep, `git checkout -- <target>` on revert — never touching anything outside the target file(s). `hooks/action_gate.py` stays active underneath: a mutation is a file edit, never a destructive op. 3. **Plateau check** every iteration (cheap, no LLM): ```bash python3 scripts/autoresearch.py plateau --store <store> --exit-code ``` Exit 10 = `--plateau-n` (default 5) consecutive reverts in a row. Do a **full rewrite** of the target from scratch (not another small nudge), then acknowledge it so the streak resets: ```bash python3 scripts/autoresearch.py record --iteration <N> --store <store> --plateau-break \ --note "plateau-break: full rewrite" ``` 4. **Health-check** — every 10th `record` call prints a `HEALTH-CHECK|` reminder line: pause and re-validate the binary criteria + validation set haven't quietly drifted before continuing.

## Step 3 — finish (squash + receipt)

```bash python3 scripts/autoresearch.py finish --store <store> --message "<Conventional Commit subject>" ```

Squashes every kept commit since the branch's start `HEAD` into ONE commit (the whole run is a SINGLE task for the DoD — the diff + the journal is the evidence, not N micro-tasks). Refuses (`BLOCKED`, exit 12) if the final kept state's gate is not `pass` — a run can never "finish" on a losing state. Writes `receipt.json` (`simplicio.savings-event/v1`):

```json { "schema": "simplicio.savings-event/v1", "source": "autoresearch", "yool_id": "agent.dev.autoresearch", "baseline": {"gate": "pass", "score": 118.0}, "actual": {"gate": "pass", "score": 71.0}, "proof": {"kind": "autoresearch-eval-log", "path": ".../journal.jsonl"}, "tokenizer": "cl100k-estimate", "iterations": 14, "kept": 4, "reverted": 9, "plateau_breaks": 1 } ```

`baseline`/`actual` are the values the eval command ITSELF produced, never hand-typed. `finish` also leaves the isolated branch checked out with the squashed commit — open the PR against it per the normal `simplicio-tasks` flow; do not merge to `main` from inside this skill.

## Other verbs

```bash python3 scripts/autoresearch.py eval --store <store> # run the eval now, print gate/score python3 scripts/autoresearch.py status --store <store> # config + plateau streak + journal tail python3 scripts/autoresearch.py selftest # pure decision math, no git/network ```

## Non-negotiables

- Never run on `main`/`master` — `init` refuses; the worker mechanically enforces isolation. - Never keep a failing gate, regardless of score — correctness before optimization, always. - Never omit `--max-iterations`/`--max-token-budget` — a cap-less loop is blocked, not "faster". - Never hand-type baseline/actual numbers in the receipt — they come from the eval command's own output, captured by the worker. - Never squash-finish on a losing state — `finish` refuses when the winner's gate isn't `pass`.

Intended uses

  • Use Simplicio Autoresearch when this documented workflow matches the task.

Implementation notes

  • Review the source instructions and adapt inputs before running the workflow.

Try it, then tell us how it went

A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.