Skip to main content
Agent skill

Dataset Curation

Prepare, format, and validate datasets for supervised fine-tuning and preference training. Use when converting raw data into training format, applying chat templates, configuring sequence packing, generating synthetic training data, or writing a dataset card before a run.

Get started

Get the complete skill folder, including its scripts and reference files.

When to use this skill

Use Dataset Curation when an AI agent needs a reusable SKILL.md workflow for this job: Prepare, format, and validate datasets for supervised fine-tuning and preference training. Use when converting raw data into training format, applying chat templates, configuring sequence packing, generating synthetic training data, or writing a dataset card before a run.

When not to use it

Skip Dataset Curation when the task is outside the writing category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.

How to install

Where will you use it?

Install scope
Claude Code folder setup
mkdir -p ~/.claude/skills/dataset-curation
# Save SKILL.md to ~/.claude/skills/dataset-curation/SKILL.md
  1. Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/dataset-curation/SKILL.md. For a shared project, commit the folder at .claude/skills/dataset-curation/ instead.
  2. Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
  3. For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.

Full install guide for Claude, Cursor, and Codex →

Try a starter prompt

After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.

Starter prompt
Use the Dataset Curation skill for this task: Use Dataset Curation when this documented workflow matches the task. Confirm any required inputs or files before making changes.

What supports this recommendation?

72/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.

Documentation
77/100
Practical detail
68/100
Source evidence
63/100
Source signals
80/100
Model compatibility evidence
  • claudeinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • chatgptinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • geminiinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • copilotinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • llamainferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • perplexityinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • mistralinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • grokinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.

What this skill does

# Dataset Curation

This skill assumes `finetuning-method-selection` already routed here — the next step is preparing data, not choosing a method. What follows: format selection by target method, the template/packing mechanics behind the most common silent training failures, rules for mixing in synthetic data without collapse, and the dataset card that closes out Phase 2 before a run starts.

**Input:** raw examples (demonstrations, preference judgments, or task prompts) plus a routing decision from `finetuning-method-selection`. **Output format:** a formatted, packed, validated JSONL dataset plus a completed dataset card — the Phase 2 artifact `/finetune` checks before launching training.

## Format Selection

| Method | Shape | Rows | |---|---|---| | SFT, single-turn | Instruct (`instruction`/`response` or `prompt`/`completion`) | ~1,000+ floor | | SFT, multi-turn | Conversation / ChatML `messages` list | ~1,000+ floor | | DPO / ORPO | Preference pair (`prompt`, `chosen`, `rejected`) | Method-dependent, see `preference-optimization` | | KTO | Unpaired (`prompt`, `completion`, `label`) | Method-dependent, see `preference-optimization` | | GRPO / RLVR | Prompt-only (`prompt` + verifier metadata) | Method-dependent, see `grpo-rlvr-training` |

- **~1,000+ rows is the recommended floor for SFT**, not a target. Below it, a handful of low-quality or duplicate examples can dominate the gradient; above it, **quality over quantity** — a smaller verified, deduplicated set beats a larger noisy one. - The ChatML shape, for orientation; the other four formats plus a ShareGPT conversion note live in `references/formats-and-templates.md`:

```json {"messages": [ {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."} ]} ```

## Chat Templates and Loss Masking

Apply the target model's chat template **before** any concatenation or packing, never after — packing raw text and templating the packed blob afterward corrupts turn boundaries, landing role markers in the wrong place relative to each example.

- **Train on assistant responses only.** Mask the loss (`-100` in the labels tensor) over system/user turns and the template's own role markers — only assistant-turn content tokens contribute to loss. - **Template/tokenizer mismatches are a top silent failure mode.** A model trained against one chat template but served or evaluated with a different one degrades without erroring. Verify the same template string used in training is applied at inference and eval time. - **Keep the dataset in `messages` shape** and let the trainer template and mask it (`assistant_only_loss=True` in current TRL) — pre-rendering to a flat text field destroys the turn boundaries masking needs. Full code sketch: `references/formats-and-templates.md`. Sanity-check before training — decode only unmasked positions; expect only assistant text:

```python keep = batch["labels"][0] != -100 print(tokenizer.decode(batch["input_ids"][0][keep])) ```

## Packing

**Without packing, 40–70% of compute is spent on padding** — variable-length examples batched at a fixed sequence length waste the gap between each example's length and the batch's max. Packing concatenates multiple examples into one sequence up to the max length, cutting most of that waste.

- **Packing changes batch semantics.** A packed sequence can contain several original examples, so "steps per epoch" and any LR schedule keyed to example count shift once packing is on — recompute schedule milestones against packed-sequence count. - **MANDATORY: decode and manually inspect 5–10 packed sequences before scaling to a full run.** Confirm example boundaries land where expected, template markers are intact per sub-example, and the loss mask is still assistant-only within each packed sequence. Not optional — packing bugs are silent (the loss curve looks normal) and only surface in eval quality, hours later:

```python for seq in packed_dataset.select(range(10)): print(tokenizer.decode(seq["input_ids"])) ```

## Synthetic Data Rules

- **Keep ≥25% real data as a collapse guard.** Training on a growing share of model-generated data without a real-data floor drives measurable quality collapse over successive generations — 25% real is the minimum that holds the line. **General-domain replay rows count toward this floor** — "real" means "not generated for this task from this student," not "human-authored." An all-synthetic-by-construction dataset can meet the ≥25% floor through replay alone (see `references/synthetic-data.md`'s Replay-Mix Construction recipe); state which rows count as "real" in the dataset card rather than leaving the floor structurally unmeetable. - **Magpie and rejection sampling are the workhorses.** Magpie extracts prompts from the model's own template prior; rejection sampling generates several candidates per prompt and keeps only the ones a filter passes. Both beat naive single-shot generation. - **Targeted, student-aware generation beats static generation by 1.3–2x sample efficiency** — aiming at the student's actual failure modes hits a quality bar with fewer filtered examples. - **Typical accept rates after filtering run 10–30%.** Plan volume accordingly — a 10,000-row target at 15% accept needs ~65,000+ raw generations. - Generation-method ranking, filter funnel, replay- mix construction, and distillation pattern: `references/synthetic-data.md`.

## The Dataset Card

Every dataset that reaches training gets a card — the required Phase 2 artifact `/finetune` checks before launching. The card is not free-form documentation; it MUST carry these fields:

- **Provenance** — where every row came from (real source(s), synthetic method(s), or both), traceable to `trace-to-training-data` output. - **Counts** — total rows, and rows per split (train/eval/held-out) if split. - **Synthetic/real ratio** — the measured ratio, checked against the ≥25% real floor above. - **Dedup method** — exact-match, semantic (embedding threshold), or both; see the filter funnel in `references/synthetic-data.md`. - **Template used** — the exact chat template string/identifier, kept consistent through inference and eval — this is what ties an `eval-harness-first` run back to the checkpoint. - **Packing config** — whether packing was used, max sequence length, and confirmation the 5–10-sequence manual inspection above was done.

A dataset missing any of these six fields isn't ready for `/finetune` — the card is a gate, not a summary written after the fact.

### Phase 2 Exit Checklist

Before handing off to `/finetune`, confirm:

1. Format matches the method (table above). 2. Template applied before concatenation. 3. Loss masked to assistant turns only. 4. 5–10 packed sequences decoded and read. 5. ≥25% real data in the final mix. 6. Dataset card complete — all six fields.

## References

- `references/formats-and-templates.md` — JSONL examples per format, current-TRL masking code, and the ShareGPT conversion note. - `references/synthetic-data.md` — generation-method ranking, filter funnel, replay-mix construction, and teacher→student distillation pattern.

Related skills: `finetuning-method-selection` routes here; `lora-qlora-recipes`, `vision-sft`, and `preference-optimization` consume the datasets this skill produces; `trace-to-training-data` is the provenance source for graded-trajectory datasets; `eval-harness-first` grades the resulting checkpoint.

Intended uses

  • Use Dataset Curation when this documented workflow matches the task.

Implementation notes

  • Review the source instructions and adapt inputs before running the workflow.

Try it, then tell us how it went

A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.