Skip to main content
Agent skill

Modelopt Day0 Release

Deterministic end-to-end driver for day-0 quantized-checkpoint releases — chains PTQ → evaluation → comparison with enforced gates between stages (the evaluation stage deploys the checkpoint itself), and returns a publish decision (ACCEPT / REGRESSION / INDETERMINATE / ANOMALOUS / INFEASIBLE). Use when the user asks to "release a model at day-0", "quantize and validate model X is within N% of baseline and tell me if it's publishable", or "run the full day-0 workflow". Do NOT use for single-stage requests — quantizing only (use modelopt-ptq), serving only (use modelopt-deployment), evaluating only (use modelopt-evaluation), or comparing two existing runs (use modelopt-compare-results).

All install options
Quick install · Personal (~/.claude/skills)Project / Cursor / Desktop →
mkdir -p ~/.claude/skills/modelopt-day0-release && curl -fsSL 'https://raw.githubusercontent.com/NVIDIA/Model-Optimizer/main/skills/modelopt-day0-release/SKILL.md' -o ~/.claude/skills/modelopt-day0-release/SKILL.md

Get the complete skill folder, including its scripts and reference files.

When to use this skill

Use Modelopt Day0 Release when an AI agent needs a reusable SKILL.md workflow for this job: Deterministic end-to-end driver for day-0 quantized-checkpoint releases — chains PTQ → evaluation → comparison with enforced gates between stages (the evaluation stage deploys the checkpoint itself), and returns a publish decision (ACCEPT / REGRESSION / INDETERMINATE / ANOMALOUS / INFEASIBLE). Use when the user asks to "release a model at day-0", "quantize and validate model X is within N% of baseline and tell me if it's publishable", or "run the full day-0 workflow". Do NOT use for single-stage requests — quantizing only (use modelopt-ptq), serving only (use modelopt-deployment), evaluating only (use modelopt-evaluation), or comparing two existing runs (use modelopt-compare-results).

When not to use it

Skip Modelopt Day0 Release when the task is outside the coding category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.

How to install

Where will you use it?

Install scope
One-line install (Claude Code)
mkdir -p ~/.claude/skills/modelopt-day0-release && curl -fsSL 'https://raw.githubusercontent.com/NVIDIA/Model-Optimizer/main/skills/modelopt-day0-release/SKILL.md' -o ~/.claude/skills/modelopt-day0-release/SKILL.md
  1. Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/modelopt-day0-release/SKILL.md. For a shared project, commit the folder at .claude/skills/modelopt-day0-release/ instead.
  2. Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
  3. For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.

Full install guide for Claude, Cursor, and Codex →

Try a starter prompt

After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.

Starter prompt
Use the Modelopt Day0 Release skill for this task: Use Modelopt Day0 Release when this documented workflow matches the task. Confirm any required inputs or files before making changes.

What supports this recommendation?

71/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.

Documentation
77/100
Practical detail
68/100
Source evidence
63/100
Source signals
72/100
Model compatibility evidence
  • claudeuntested

    No model-specific signal or recorded compatibility test was found.

  • chatgptuntested

    No model-specific signal or recorded compatibility test was found.

  • geminiuntested

    No model-specific signal or recorded compatibility test was found.

  • copilotuntested

    No model-specific signal or recorded compatibility test was found.

  • llamauntested

    No model-specific signal or recorded compatibility test was found.

  • perplexityuntested

    No model-specific signal or recorded compatibility test was found.

  • mistraluntested

    No model-specific signal or recorded compatibility test was found.

  • grokuntested

    No model-specific signal or recorded compatibility test was found.

Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.

What this skill does

# Day-0 Release

Drive a model from a pretrained checkpoint to a publish decision for a quantized checkpoint, in a fixed sequence with a gate after every stage. This skill is a **conductor**: it sequences the existing domain skills and enforces the gates — it does not re-implement quantization, serving, evaluation, or comparison.

**Goal (the default day-0 criterion):** a quantized checkpoint smaller than the source, with accuracy drop within the threshold (default <1%) on the standard benchmark set versus the matching baseline, plus a publish recommendation.

## When to use

Use only for the full goal-driven release. For a single stage, route to the domain skill directly: quantize → **ptq**, serve → **deployment**, evaluate → **evaluation**, compare two existing runs → **compare-results**.

## Inputs

Resolve these before starting (ask the user for anything missing):

- **Model** — HF handle or checkpoint path. - **Recipe / qformat** — e.g. `nvfp4`, `fp8`, or a recipe path. One candidate for v1. - **Cluster / launcher** — from `clusters.yaml` (see the common skill's `environment-setup.md`). - **Eval set** — defaults to the evaluation skill's AA suite (`recipes/tasks/aa/`). - **Threshold** — max accuracy drop; default `0.01` (1%).

## The chain

```text setup ─▶ PTQ ─▶ canary ─▶ baseline-eval ─▶ quantized-eval ─▶ compare ─▶ verbosity ─▶ closeout │ │ │ │ │ │ gate_ptq /health gate_run gate_run gate_compare gate_verbosity + 1 gen ```

The **evaluation** skill deploys the model it evaluates (it stands up its own endpoint per run), so there is no separate deploy *stage* — a serving failure during evaluation surfaces through the eval gate (`DEPLOYMENT_HEALTH_FAILED`) and triages to the **deployment** skill (see Step 4). The Step 2b **canary** is not that: it runs *before* any evaluation precisely so an unservable checkpoint is caught in ~15 min rather than after a multi-hour eval.

Accuracy (Step 5) and verbosity (Step 5b) are **independent gates**; closeout requires both.

Run each stage by invoking the domain skill, then run its gate before proceeding. **Do not advance past a failed gate.** Copy this checklist and track progress:

```text - [ ] Step 0: Resolve inputs; confirm threshold and eval set - [ ] Step 1: Setup gate — creds present, cluster reachable - [ ] Step 2: PTQ (ptq skill) → gate_ptq.py - [ ] Step 2b: Serving canary — /health + one generation (deployment skill) - [ ] Step 3: Baseline eval (evaluation skill, deploys source) → gate_run.py - [ ] Step 4: Quantized eval (evaluation skill, deploys candidate) → gate_run.py - [ ] Step 5: Compare (compare-results skill) → external sanity → gate_compare.py → decision - [ ] Step 5b: Verbosity gate → gate_verbosity.py - [ ] Step 6: Closeout — report + publish recommendation ```

### Step 1 — Setup gate

Use the common skill's `credentials.md` and `remote-execution.md` to confirm credentials and cluster reachability. If either fails, stop with `SYSTEMIC` — do not start PTQ.

### Step 2 — PTQ

Invoke the **ptq** skill to produce the quantized checkpoint. Then gate:

```bash # The ptq skill's post-PTQ validation produces a validation-summary JSON (size # ratio + layer-precision counts + metadata diffs; see the ptq skill's # references/checkpoint-validation.md). v1 gates on that summary: python "$SKILL_DIR/scripts/gate_ptq.py" --summary <validation-summary.json> # add `--recipe <qformat>` to override the recipe recorded in the summary ```

`gate_ptq.py` returns JSON `{pass, failure_class, detail}`. On `pass: false`, branch on `failure_class` (see **Triage** below). Do not evaluate an unvalidated checkpoint.

#### Step 2b — serving canary (MANDATORY before Step 3)

The canary itself is **already specified** by the ptq skill: see `modelopt-ptq/references/checkpoint-validation.md` (required gate, canary query and the `Serving canary` row of its report table) and `modelopt-ptq/SKILL.md`. Run it there rather than re-deriving it here — `gate_ptq.py` checks size, coverage and metadata, not whether the checkpoint *loads*, and skipping the canary has cost a full baseline eval against a checkpoint the serving stack could never load.

On failure use `failure_class: CHECKPOINT_NOT_SERVABLE` and drop to the **deployment** skill; do not proceed to Step 3.

**What that spec does not cover: writing a canary that cannot lie.** Both of these produced a wrong verdict on a large MoE, in opposite directions:

- **Poll ceiling > load time**, with headroom. A 50 min poll against a 51 min load reported failure for a checkpoint that serves fine. Large MoE loads are CPU-bound fp8 dequant (~50 min); 0% GPU during *load* is normal. - **Print an explicit `RESULT:` on every path** and exit non-zero on failure. A poll loop that falls through to the generation test exits 0 and reads as PASS. - Server log on shared storage, not node-local `/tmp`. - Canary the **as-exported** artifact, not a copy you modified to make it work.

### Step 3 — Baseline eval

The baseline is the **source** (pre-quantization) model on the same task set and sampling params. Always run a fresh baseline via the **evaluation** skill, which deploys the source model itself. Gate every run with `gate_run.py`.

Before submitting either side, record per-task run counts, uncertainty method, and stopping rule. For retry-level Welch, plan at least two fresh independent full-benchmark runs per side (SciCode: at least eight), increasing these fixed counts up front if more precision is needed. One AA-suite invocation does not supply multiple full runs for GPQA or IFBench: in-run repeats are not independent retry scores. A task-specific method using repeat/trial data instead must justify its observation unit and dependence assumptions.

Complete the declared plan before the final assessment. Result-dependent reruns, repeated looks, or selecting candidates on these results are exploratory: require independent confirmation under a fresh declared plan, or a justified sequential/selection-aware method, before confidence-based ACCEPT.

### Step 4 — Quantized eval

Invoke the **evaluation** skill on the quantized checkpoint, matching the baseline's task set, sampling params, and declared run plan. The evaluation skill stands up the serving endpoint itself (it builds the `deployment.command`, e.g. a `vllm serve …`), so a serving failure surfaces here as a failed `gate_run.py` with `DEPLOYMENT_HEALTH_FAILED`. When that happens, **drop to the deployment skill** to reproduce and debug serving in isolation (serve the checkpoint standalone, confirm `/health` + one generation, iterate on flags / TP / image / env vars) rather than burning full eval cycles on a broken endpoint — then carry the working command back into NEL's `deployment.command` and resume the eval. If the checkpoint genuinely can't serve, `POINT_INFEASIBLE`.

**Before submitting: assert baseline/candidate config parity.** The candidate config must differ from the baseline's in nothing but checkpoint path and served model name. Diff mechanically — an eyeball pass misses this:

```bash diff <(grep -vE 'checkpoint_path|served_model_name' baseline.yaml) \ <(grep -vE 'checkpoint_path|served_model_name' candidate.yaml) ```

Any other difference biases the comparison and invalidates the gate: a mismatched `parallelism` between the two sides was worth ~2 pp, enough to invert the sign of the delta. If the model card splits sampling params per scenario, apply the same split to both sides. Never set an unbounded `request_timeout` (`1e9`) — it turns a transient stall into a job that holds its GPUs until the wall clock kills it.

Gate:

```bash python "$SKILL_DIR/scripts/gate_run.py" --run <run-summary.json> ```

A `pass: false` here means the run is incomplete or invalid (judge/parse error, dropped samples) — do **not** compare scores from it.

### Step 5 — Compare

Invoke the **compare-results** skill. It must perform the shared external baseline sanity check before the candidate-delta gate. A failed check is `ANOMALOUS` with failure class `EXTERNAL_BASELINE_MISMATCH`: investigate and rerun the baseline. If no credible comparable external score exists, record the baseline as externally unverified and continue using the validated measured baseline.

#### Uncertainty — check BEFORE trusting any per-task verdict

Use the [compare-results assessment](../modelopt-compare-results/references/assess-results.md) against the allowed loss, in matching units. A point estimate within threshold or an SE below it does not establish acceptance; require the entire signed-loss confidence interval above the minimum acceptable margin.

Apply Step 3's declared run/stopping plan.

**Re-running does not guarantee fresh samples.** With a warm NEL response cache a "re-run" can replay cached responses — two runs came back bit-identical to 16 digits. But an equal score is a signal to check provenance (invocation id, output dir, response artifacts), **not** proof of replay: discrete-scored tasks tie legitimately. Confirm replay before discarding a run as a duplicate.

After recording the external status, produce per-task deltas and run:

```bash python "$SKILL_DIR/scripts/gate_compare.py" \ --baseline <baseline_scores.json> --candidate <candidate_scores.json> \ --threshold 0.01 ```

The threshold is a fraction of each task's score scale. Most AA tasks report 0-100, but some (e.g. `tau2_bench_telecom` `Result`) report 0-1; the gate infers each task's scale (0-1 if both scores are within [0, 1], else 0-100) and normalizes the drop accordingly, so `--threshold 0.01` means "≤1 pt on a 0-100 task / ≤0.01 on a 0-1 task" uniformly. Pass `--scales '{"task": max}'` to override inference if a task's scores happen to fall in an ambiguous range.

`gate_compare.py` is a preliminary point-estimate check, not the final accuracy verdict. It cannot override external sanity or the interval assessment. Combined decision (external sanity failures and implausible scores take precedence):

- **ACCEPT (accuracy only)** — no external check failed and every task's assessment is **Within margin** → continue to **Step 5b**. Advance to Step 6 only once the verbosity gate also passes. A missing comparable external score is not a failure; report it as externally unverified. - **REGRESSION** — at least one task's assessment is **Substantial regression**. **v1 stops here and reports** which tasks regressed by how much. - **INDETERMINATE** — otherwise, any task's target assessment is unresolved or unavailable (including insufficient valid runs). Stop with `do-not-publish`; report the missing evidence rather than accepting the point-estimate verdict. - **ANOMALOUS** — external baseline sanity failed, or scores are otherwise implausible (e.g. baseline lower than candidate by a large margin, or a task score is outside its valid range) → correct the baseline or surface it.

#### Step 5b — Verbosity gate (MANDATORY; independent of accuracy)

`gate_compare.py` does not measure verbosity, so stopping after Step 5 leaves a hard gate unmeasured. Step 5's ACCEPT is accuracy-only — Step 6 requires both.

```bash python "$SKILL_DIR/scripts/gate_verbosity.py" \ --baseline <baseline_eval_root> --candidate <candidate_eval_root> \ --glob 'eval_*' --exclude _high --threshold 0.05 ```

Exit codes follow the other gates: `0` pass, `1` the gate failed, `2` it could not read its input. A `2` means fix the invocation, not the checkpoint — check `--baseline`/`--candidate`, `--glob` (`no_files_matched` names the pattern that missed), and `--exclude`. If the detail names `collapsed_keys`, the artifact tree gave several tasks one name: point the gate at a tree whose task dirs are `<harness>.<task>` rather than `<invocation_id>.<job_index>`, or re-sync so each `artifacts/` carries its own `config.yml`.

Read `response_stats.avg_completion_tokens` from each task's `artifacts/eval_factory_metrics.json`. Do **not** use the `reasoning.*` fields: `reasoning.*_tokens` are always `0` (only the reasoning/content split is missing) and the `*_words` siblings are a proxy that can disagree with the gate — one task read +6.10% FAIL in words and +1.32% PASS in tokens.

The gate is two-sided; a large drop in output length is also a change.

Two filters are mandatory: same reasoning effort (pass `--exclude` explicitly — it is empty by default, since which tier is canonical is run-specific) and complete runs only — runs within 1% of the largest `successful_count` that **both** sides can match (exact equality would make one dropped sample unmeasurable). A task with no such count on both sides is `not_comparable`. When the matched count is below a run one side has, the task carries `truncated_comparison`.

Tasks sharing no sample count are `not_comparable`, not a delta. Means under ~1000 tokens carry a `short_output_warning` — read absolute counts there.

### Step 6 — Closeout

Report the decision with: source vs output size + ratio, per-task baseline / candidate / signed-loss CI / margin / assessment, **the verbosity verdict from Step 5b**, external source and sanity status, MLflow run IDs, and a publish recommendation (publish / do-not-publish). Archive artifacts to the workspace.

**Publish exactly what was evaluated.** Verify mechanically, not by path convention: inode-compare a shard in the evaluated directory against the one being published. A day-0 run leaves near-identical sibling exports that differ by one calibration suffix — prefix rejected ones `REJECTED-` so the artifact cannot be picked by autocomplete, and re-run the Step 2b canary against the final path after any move.

## Triage (gate failure → decision)

For completed evaluations, apply [evaluation run validation](../modelopt-evaluation/references/run-validation.md) before triage; its policy overrides retry actions below. Warn but do not retry solely for failures below tolerance. For runs that fail tolerance or an independent validation check, return findings and a recommendation to the parent/user for review; do not automatically resubmit.

Map a gate's `failure_class` to the next action:

| `failure_class` | Action | | --- | --- | | `INFRA_TRANSIENT` | Retry the stage once; if it recurs, `SYSTEMIC`. | | `MODEL_UNSUPPORTED` | PATCH: fix the recipe pattern / add model support (ptq skill owns the patch loop), then retry. If unpatchable, `POINT_INFEASIBLE`. | | `QUANT_COVERAGE_FAILURE` | PATCH: fix the recipe wildcard so intended layers are covered; re-run PTQ. | | `SIZE_NOT_REDUCED` | The output is not smaller than the source. If the recipe cannot shrink the source (e.g. mxfp4 under nvfp4, or fp8 under fp8), record `source_precision` in the validation summary and re-run the gate — that states *why* the growth is expected. Otherwise treat it as a real compression failure: check that the recipe matched the intended parameter mass and that the exporter did not retain the original tensors. `accept_size_growth: true` waives it unconditionally (no growth bound) and is a last resort, not the fix — it records no reason, so prefer `source_precision` where the claim is checkable. | | `CHECKPOINT_NOT_SERVABLE` | The Step 2b canary could not load/generate. Usually a tensor-naming or config-schema mismatch between the exporter and the serving stack, or missing/dangling auxiliary files (tokenizer). Fix the export; do not evaluate. | | `VERBOSITY_EXCEEDED` | Re-check run hygiene first (mixed reasoning effort, partial runs, unequal sample counts) — that has explained every false positive so far. If the delta survives matched, complete runs, it is a real behavioural change; do not publish on accuracy alone. | | `DEPLOYMENT_HEALTH_FAILED` | Drop to the **deployment** skill: reproduce serving standalone (`/health` + one generation), debug flags / image / TP / env, then carry the working command into NEL's `deployment.command` and retry the eval. If it can't serve, `POINT_INFEASIBLE`. | | `EVAL_JUDGE_FAILED` | For completed runs, apply evaluation run validation above; return tolerance failures to the parent/user without automatic retry. Otherwise investigate auth/rate limits before retrying. | | `SAMPLE_ACCOUNTING_FAILED` | Investigate dropped/failed samples before trusting scores. | | `EXTERNAL_BASELINE_MISMATCH` | Investigate baseline configuration, correct it, rerun the baseline, and repeat external sanity before comparison. | | `USER_CONFIG_ERROR` | Correct it from the request, workspace, or model/config metadata and retry; if irrecoverable, return `ANOMALOUS` with evidence. | | `UNKNOWN` | Investigate with the owning domain skill; if unresolved, return `ANOMALOUS` with the evidence and next automated retry or patch action. |

`gate_ptq.py` also emits non-blocking `notes` (present on every result). Size growth is **blocking by default**, waived to a note only when the validation summary's `source_precision` is already at or below the recipe's target bits and the growth is within what that explains. Recording `source_precision` is part of the ptq skill's validation table, so the waiver is reachable from the normal pipeline. A BF16 source that failed to compress still fails, which is the case this check exists for.

`SYSTEMIC` (cluster down, dataset unavailable) aborts the whole run. `POINT_INFEASIBLE` means this (model, recipe) can't work as configured.

## Output

Return a decision, not a raw artifact:

- `ACCEPT` + report + publish recommendation - `REGRESSION` + which tasks failed the threshold and by how much - `INDETERMINATE` + unresolved tasks, missing evidence, and `do-not-publish` - `ANOMALOUS` / `INFEASIBLE` + reason and next automated action - Always: workspace path + MLflow run IDs for traceability

## Scope (v1)

In v1: the linear chain + gates + report. On `REGRESSION`, v1 reports and stops. Deferred to a follow-up: the evaluator-optimizer recipe loop (compare → pick the next recipe → re-run PTQ), which needs the bigpareto integration and a shared config/result schema.

Intended uses

  • Use Modelopt Day0 Release when this documented workflow matches the task.

Implementation notes

  • Review the source instructions and adapt inputs before running the workflow.

Try it, then tell us how it went

A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.