Trace To Training Data
Convert evaluation traces and production logs into SFT examples and preference pairs. Use when graded traces or failure examples exist and need to become training data, when applying rejection sampling to model outputs, or when building DPO pairs from passing and failing runs.
Get the complete skill folder, including its scripts and reference files.
When to use this skill
Use Trace To Training Data when an AI agent needs a reusable SKILL.md workflow for this job: Convert evaluation traces and production logs into SFT examples and preference pairs. Use when graded traces or failure examples exist and need to become training data, when applying rejection sampling to model outputs, or when building DPO pairs from passing and failing runs.
When not to use it
Skip Trace To Training Data when the task is outside the coding category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.
How to install
Where will you use it?
mkdir -p ~/.claude/skills/trace-to-training-data # Save SKILL.md to ~/.claude/skills/trace-to-training-data/SKILL.md
- Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/trace-to-training-data/SKILL.md. For a shared project, commit the folder at .claude/skills/trace-to-training-data/ instead.
- Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
- For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.
Full install guide for Claude, Cursor, and Codex →
Try a starter prompt
After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.
Use the Trace To Training Data skill for this task: Use Trace To Training Data when this documented workflow matches the task. Confirm any required inputs or files before making changes.
What supports this recommendation?
63/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.
- Documentation
- 67/100
- Practical detail
- 50/100
- Source evidence
- 63/100
- Source signals
- 80/100
Model compatibility evidence
- claudeinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- chatgptinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- geminiinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- copilotinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- llamainferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- perplexityinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- mistralinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- grokinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.
What this skill does
# Trace To Training Data
This skill assumes `eval-harness-first` already graded the traces being converted here — goldens, graders, and `runs/<run-id>/results.json` all exist before conversion starts. This is the flywheel edge that skill names in its own flow: "the same labeled traces become the training set." Conversion happens here; grading already happened upstream.
**Input:** graded traces — `eval/goldens.jsonl` plus `runs/<run-id>/results.json`, each row carrying a `task_id`, a `verdict` from the grader, and a `reward` when the task supports a scalar score (judge score, execution partial-credit, or an RLVR verifier):
```json {"task_id": "t-042", "trace_id": "t-042-a3", "messages": [{"role": "user", "content": "..."}], "verdict": "pass", "reward": 0.91, "grader": "exact_match"} ```
**Output format:** rows shaped exactly like `dataset-curation`'s Format Selection table — SFT `messages` rows or DPO `prompt`/`chosen`/`rejected` pairs — so this skill's output is that skill's input with no reshaping step in between.
## The Principle
The eval harness already did the labeling work: every trace in `results.json` carries a verdict, and often a reward, before this skill ever touches it. Converting a graded trace into a training row is mechanical — pick a shape from `dataset-curation`'s table, map fields, write JSONL. **Curation is the work that remains** — which traces clear a quality bar, which pairs are informative, and which rows must never enter the training set at all.
Treat any conversion step that requires re-judging a trace as a sign the harness is missing a grader, not a gap this skill should paper over. A trace with no verdict or reward isn't convertible yet — route it back to `eval-harness-first` first, don't hand-label it here to unblock conversion.
## SFT From Traces
- **Keep the top-reward fraction of successful trajectories**, not every passing one. Rank passing traces by reward and take a fraction (the Agent-lightning pattern) rather than every trace that merely cleared the pass bar — a trace that barely passed is a weaker SFT signal than one that scored well above threshold. - **Expert-corrected failures become gold SFT examples directly** (the Langfuse pattern) — when a human edits a failing trace's output into a correct one, that correction needs no reward threshold; a human already validated it. Route corrections straight into the SFT set. - **Step-level masking beats whole-trajectory discard for multi-step traces.** When only some steps in a multi-step trajectory are bad, mask the loss on the bad steps and keep the good ones, rather than discarding the whole trajectory. SRFT reports 32.2% vs. 30.9% on SWE-bench for step-level critic masking over trajectory discard — a real, if modest, gap from the finer-grained cut.
## Preference Pairs From Traces
- **Build pairs from passing-vs-failing trajectories on the SAME task**, never from unrelated best- and worst-scoring traces pulled across different tasks — cross-task pairs teach the model to prefer one task over another, not one response over another. - **Select the rejected member at μ−2σ of the reward distribution for that task, never the absolute minimum.** `preference-optimization`'s Pair Construction section owns the full selection formula; this skill supplies the graded trajectories it consumes. - **Judge-scored delta selection cuts pair volume without cutting signal.** Score each candidate pair by chosen-minus-rejected judge delta and keep only the highest-delta subset — the top 5k of a 16.5k candidate pool matched the full pool's downstream result. Build the full candidate set first, then filter by delta; don't cap generation at 5k up front.
## Hygiene
- **Scan for secrets and PII before any row ships, and redact what's found.** Traces sourced from production logs can carry credentials, API keys, tokens, or customer data — run a secret/PII scan over every SFT and DPO row and redact matches; conversion fails closed (the row is dropped, not shipped with the raw content) if sensitive fields remain after redaction. Never commit secrets. - **Eval goldens must never leak into training data.** Hold every `eval/goldens.jsonl` ID out of every converted SFT and DPO set — a trace that also appears as a golden trains on the exact item the checkpoint gets graded against later, silently inflating every subsequent eval run. - **Dedup against the training set**, not just within the newly converted rows — exact-match or embedding-similarity, matching `dataset-curation`'s dedup method field, run against whatever training data already exists before this batch merges in. - **Provenance goes into the dataset card.** Every converted row must trace back to its source `run_id` and `trace_id` — `dataset-curation`'s Provenance field checks for exactly this link back to `trace-to-training-data` output; a row with no traceable source isn't ready to merge.
## Related Skills
- `eval-harness-first` — produces the graded traces this skill converts; a trace with no verdict or reward isn't convertible yet, route it back there before conversion. - `dataset-curation` — owns the target formats and the dataset card this skill's provenance data feeds; converted rows must match its Format Selection table field names exactly, not an approximation of them. - `preference-optimization` — consumes the DPO pairs this skill builds and owns the full μ−2σ rejection-selection formula referenced above.
Worked JSONL-to-JSONL conversions — graded trace to SFT row, trace pair to DPO pair, correction to SFT row, the rejection-sampling loop, and the goldens-holdout check — live in `references/conversion-recipes.md`.
Intended uses
- Use Trace To Training Data when this documented workflow matches the task.
Implementation notes
- Review the source instructions and adapt inputs before running the workflow.
Try it, then tell us how it went
A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.