Skip to main content
Agent skill

Modelopt Evaluation

Evaluates accuracy of quantized or unquantized LLMs using NeMo Evaluator Launcher (NEL). Triggers on "evaluate model", "benchmark accuracy", "run MMLU", "evaluate quantized model", "run nel". Handles deployment, config generation, and evaluation execution. Always load when validating completed runs or reporting evaluation scores, including standalone modelopt-launching-evals analysis. Not for quantizing models (use modelopt-ptq), standalone serving (use modelopt-deployment), or comparing completed baseline-vs-quantized results (use modelopt-compare-results).

All install options
Quick install · Personal (~/.claude/skills)Project / Cursor / Desktop →
mkdir -p ~/.claude/skills/modelopt-evaluation && curl -fsSL 'https://raw.githubusercontent.com/NVIDIA/Model-Optimizer/main/skills/modelopt-evaluation/SKILL.md' -o ~/.claude/skills/modelopt-evaluation/SKILL.md

Get the complete skill folder, including its scripts and reference files.

When to use this skill

Use Modelopt Evaluation when an AI agent needs a reusable SKILL.md workflow for this job: Evaluates accuracy of quantized or unquantized LLMs using NeMo Evaluator Launcher (NEL). Triggers on "evaluate model", "benchmark accuracy", "run MMLU", "evaluate quantized model", "run nel". Handles deployment, config generation, and evaluation execution. Always load when validating completed runs or reporting evaluation scores, including standalone modelopt-launching-evals analysis. Not for quantizing models (use modelopt-ptq), standalone serving (use modelopt-deployment), or comparing completed baseline-vs-quantized results (use modelopt-compare-results).

When not to use it

Skip Modelopt Evaluation when the task is outside the analysis category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.

How to install

Where will you use it?

Install scope
One-line install (Claude Code)
mkdir -p ~/.claude/skills/modelopt-evaluation && curl -fsSL 'https://raw.githubusercontent.com/NVIDIA/Model-Optimizer/main/skills/modelopt-evaluation/SKILL.md' -o ~/.claude/skills/modelopt-evaluation/SKILL.md
  1. Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/modelopt-evaluation/SKILL.md. For a shared project, commit the folder at .claude/skills/modelopt-evaluation/ instead.
  2. Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
  3. For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.

Full install guide for Claude, Cursor, and Codex →

Try a starter prompt

After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.

Starter prompt
Use the Modelopt Evaluation skill for this task: [ ] Step 0: Check workspace (if MODELOPTWORKSPACEROOT set). Confirm any required inputs or files before making changes.

What supports this recommendation?

74/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.

Documentation
82/100
Practical detail
74/100
Source evidence
63/100
Source signals
72/100
Model compatibility evidence
  • claudeuntested

    No model-specific signal or recorded compatibility test was found.

  • chatgptuntested

    No model-specific signal or recorded compatibility test was found.

  • geminiuntested

    No model-specific signal or recorded compatibility test was found.

  • copilotuntested

    No model-specific signal or recorded compatibility test was found.

  • llamauntested

    No model-specific signal or recorded compatibility test was found.

  • perplexityuntested

    No model-specific signal or recorded compatibility test was found.

  • mistraluntested

    No model-specific signal or recorded compatibility test was found.

  • grokuntested

    No model-specific signal or recorded compatibility test was found.

Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.

What this skill does

# NeMo Evaluator Launcher Assistant

Guide the user through creating NEL YAML configs, running evaluations, and monitoring progress. NEL manages serving for canaries and full runs; see [Step 3's serving-handoff check](references/launcher-workflow.md#step-3--configure-deployment) for external endpoints.

## Completed-run analysis

For completed runs, skip launch/config; use launching-evals for artifacts/analysis and apply [run-validation.md](references/run-validation.md) before scores. Its policy overrides blanket uncapping and fixed token/context limits.

## Workspace integration

If `MODELOPT_WORKSPACE_ROOT` is set, use the common skill's `workspace-management.md` and reuse existing workspaces (PTQ → checkpoint smoke test → NEL evaluation; carry compatibility requirements into NEL's deployment config).

## Workflow

```text - [ ] Step 0: Check workspace (if MODELOPT_WORKSPACE_ROOT set) - [ ] Step 1: Check `nel` install + existing config; set up `.env` (+ `modelopttools:eval-config` for judge-scored runs) - [ ] Step 2: Build base config (5-question flow OR shortcut) - [ ] Step 3: Configure deployment (model path, params, cross-check) - [ ] Step 4: Fill remaining ??? values - [ ] Step 5: Confirm tasks (iterative) - [ ] Step 6: Multi-node (if needed) - [ ] Step 7: Interceptors (if needed) - [ ] Step 7.5: Container auth (SLURM private images) - [ ] Step 8: Dry-run → canary → full run - [ ] Step 9: Verify completed run ```

---

## nel-next path (Terminal-Bench 2.x, SWE-bench, …) — branch here FIRST

A few **agentic** AA benchmarks do **not** run on the currently validated `nemo-evaluator-launcher` 0.2.6 path (Steps 1–9 don't apply). They run on **nel-next** (`nemo-evaluator[harbor]` 0.4.x) — a separate package, CLI (`nel eval run`), `-O` overrides, and `services`/`benchmarks`/`cluster`/`output` schema. If the user asks for one, do **not** add it to a 0.2.6 `evaluation.tasks` list — instead:

1. Read **`references/nel-next.md`** (shared: venv, schema, AWS creds, architecture, timeout strategy, MLflow, run flow) + the per-benchmark recipe `recipes/tasks/aa_next/{terminal_bench_2_1,swebench_verified}.md`; start from `recipes/examples/example_eval_next.yaml`. Read [model-card research](references/model-card-research.md) for generation settings; consult task-specific policy before selecting published budgets or fallbacks. 2. Isolated nel-next venv: `"$SKILL_DIR/scripts/nel-next.sh" --setup-only` (keeps 0.2.6 `nel` untouched). 3. Run **`modelopttools:eval-config`** (Step 3b) to write the AWS-sandbox creds + harbor infra rows (`${NEL_NEXT_EVAL_IMAGE}`, `${HARBOR_*_ECR_REPOSITORY}`, `${HARBOR_ECS_REGION}`) into `.env`; always include the `output.export_config.mlflow` block. 4. Dry-run → canary → full (`nel-next.sh eval run`), then **push to MLflow** — SLURM doesn't auto-export, so run `nel-next.sh mlflow-push -r <run_id> -c <cfg>` after (config-driven; see `references/nel-next.md`). 5. Before reporting a full-run score, apply `references/run-validation.md`, including its benchmark-specific checks.

Steps 1–9 below are currently validated with 0.2.6 — use them for everything else.

---

## MRCR (NeMo Gym `simple_agent`) path — branch here too

MRCR **does** run on the currently validated 0.2.6 `nel` launcher (as a `nemo_gym` task, not nel-next), so Steps 1–9 apply — but it is mechanically special and **standalone** (one gym eval per config; never mix it with `aa/` tasks). It is simple as gym tasks go: `simple_agent`, **no judge** — deterministic prefix-gated grading, `HF_TOKEN` the only secret. **Not an AA benchmark** — never generate it for an "AA" request. If the user asks for MRCR:

1. Read **`references/gym.md`** (pinned launcher, gym prepare/reap machinery, pin↔container coupling, preflight gaps, failure modes) + **`recipes/tasks/gym/mrcr.md`**; start from **`recipes/examples/gym/example_mrcr.yaml`** (1M variant, like the golden). 2. **Pick the variant first** (`config_n3_1m` / `config_n3_128k` / `config`) — it sets the context cap, dataset *and* metric prefix; the three are not comparable; set it in **both** `data_prep_params` and `collect_rollout_params`. Take the largest variant within the checkpoint's trained context — see the recipe. 3. `.env`: `HF_TOKEN` (dataset + n3 tokenizer are gated) plus `NEMO_EVALUATOR_TRUST_PRE_CMD=1` (the `pre_cmd` installs `tiktoken` + `transformers`; prepare fails without it) and `NEMO_EVALUATOR_TRUST_UNLISTED_TASKS=1` (`nemo_gym` is not in the FDF map). 4. **MRCR needs a git-backed Gym image.** The pin is newer than any image's baked Gym and must apply, so the template's `container:` is `???` and the bootstrap exits 1 on a non-git `/opt/Gym` (the public `eval-factory/nemo-gym:*` images). NVIDIA-internal: `modelopttools:eval-config` Step 3d names a working image. 5. Long-context deploy: for **1M**, `--max-model-len 1100000` + `VLLM_ALLOW_LONG_MAX_MODEL_LEN=1`; for **128K**, `--max-model-len` at the checkpoint's trained context (room for the answer above 131,072) and no override. 1M also needs `gpu_memory_utilization: 0.95`. Both: multi-instance fan-out; **never cap output tokens**; report the needle-count strata alongside `pass@1/accuracy`. 6. Run both dry-run and launch through `"$SKILL_DIR/scripts/nel-gym.sh"`; it enforces the currently validated 0.2.6 launcher even if `nel` on PATH is stale and avoids an unset `NEL_INVOCATION_ID` failure before client startup. Canary with `limit_samples` — the MRCR template forwards it as `++limit` (see the recipe's Canary section); the prepare pass still runs in full.

---

Detailed launcher instructions live in [launcher-workflow.md](references/launcher-workflow.md). Read only the sections needed for the current stage; retain the dry-run → canary → full-run gates. Existing configs must pass Step 3's serving-handoff check before Step 8. Paths in that reference are relative to this skill directory.

## Step 1 — Prerequisites

Read [Step 1](references/launcher-workflow.md#step-1--prerequisites).

## Step 2 — Build base config (when not using shortcut)

Read [Step 2](references/launcher-workflow.md#step-2--build-base-config-when-not-using-shortcut).

## Step 3 — Configure deployment

Read [Step 3](references/launcher-workflow.md#step-3--configure-deployment).

## Step 4 — Fill remaining ??? values

Read [Step 4](references/launcher-workflow.md#step-4--fill-remaining--values).

## Step 5 — Confirm tasks (iterative)

Read [Step 5](references/launcher-workflow.md#step-5--confirm-tasks-iterative).

## Step 6 — Multi-node

Read [Step 6](references/launcher-workflow.md#step-6--multi-node).

## Step 7 — Interceptors

Read [Step 7](references/launcher-workflow.md#step-7--interceptors).

If the user asks to avoid doom loops (looping reasoning), or a canary under a long `max_new_tokens` runs far longer than expected, read [doom_loops.md](references/doom_loops.md).

## Step 7.5 — Container registry auth (SLURM private images only)

Read [Step 7.5](references/launcher-workflow.md#step-75--container-registry-auth-slurm-private-images-only).

## Step 8 — Run evaluation (gated dry-run → canary → full)

Read [Step 8](references/launcher-workflow.md#step-8--run-evaluation-gated-dry-run--canary--full).

## Step 9 — Verify completed run

Apply [run-validation.md](references/run-validation.md)'s **Bounded Evaluation-Failure Policy** and **Timeout and Output-Limit Accounting** before reporting scores. For comparisons, also apply its **External Baseline Sanity Check**, then use `modelopt-compare-results`.

---

Issues: <https://github.com/NVIDIA-NeMo/Evaluator/issues> · <https://github.com/NVIDIA-NeMo/Evaluator/discussions>

Intended uses

  • [ ] Step 0: Check workspace (if MODELOPTWORKSPACEROOT set)
  • [ ] Step 1: Check nel install + existing config; set up .env (+ modelopttools:eval-config for judge-scored runs)
  • [ ] Step 2: Build base config (5-question flow OR shortcut)
  • [ ] Step 3: Configure deployment (model path, params, cross-check)
  • [ ] Step 4: Fill remaining ??? values
  • [ ] Step 5: Confirm tasks (iterative)
  • [ ] Step 6: Multi-node (if needed)
  • [ ] Step 7: Interceptors (if needed)
  • [ ] Step 7.5: Container auth (SLURM private images)
  • [ ] Step 8: Dry-run → canary → full run
  • [ ] Step 9: Verify completed run

Implementation notes

  • Review the source instructions and adapt inputs before running the workflow.

Try it, then tell us how it went

A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.