Modelopt Launching Evals
Run, monitor, analyze, and debug LLM evaluations via nemo-evaluator-launcher. Covers running evaluations, checking status and live progress, debugging failed runs, exporting artifacts and logs, and analyzing results. ALWAYS triggers on mentions of running evaluations, checking progress, debugging failed evals, analyzing or analysing runs or results, run directories or artifact paths on clusters, Slurm job issues, invocation IDs, or inspecting logs (client logs, server logs, SSH to cluster, tail logs, grep logs). Do NOT use for creating or modifying evaluation configs.
mkdir -p ~/.claude/skills/modelopt-launching-evals && curl -fsSL 'https://raw.githubusercontent.com/NVIDIA/Model-Optimizer/main/skills/modelopt-launching-evals/SKILL.md' -o ~/.claude/skills/modelopt-launching-evals/SKILL.md
Get the complete skill folder, including its scripts and reference files.
When to use this skill
Use Modelopt Launching Evals when an AI agent needs a reusable SKILL.md workflow for this job: Run, monitor, analyze, and debug LLM evaluations via nemo-evaluator-launcher. Covers running evaluations, checking status and live progress, debugging failed runs, exporting artifacts and logs, and analyzing results. ALWAYS triggers on mentions of running evaluations, checking progress, debugging failed evals, analyzing or analysing runs or results, run directories or artifact paths on clusters, Slurm job issues, invocation IDs, or inspecting logs (client logs, server logs, SSH to cluster, tail logs, grep logs). Do NOT use for creating or modifying evaluation configs.
When not to use it
Skip Modelopt Launching Evals when the task is outside the coding category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.
How to install
Where will you use it?
mkdir -p ~/.claude/skills/modelopt-launching-evals && curl -fsSL 'https://raw.githubusercontent.com/NVIDIA/Model-Optimizer/main/skills/modelopt-launching-evals/SKILL.md' -o ~/.claude/skills/modelopt-launching-evals/SKILL.md
- Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/modelopt-launching-evals/SKILL.md. For a shared project, commit the folder at .claude/skills/modelopt-launching-evals/ instead.
- Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
- For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.
Full install guide for Claude, Cursor, and Codex →
Try a starter prompt
After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.
Use the Modelopt Launching Evals skill for this task: Create or modify a config using the nel-assistant skill. If the user provides a past run, use its config.yml artifact as a starting point. Confirm any required inputs or files before making changes.
What supports this recommendation?
74/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.
- Documentation
- 82/100
- Practical detail
- 74/100
- Source evidence
- 63/100
- Source signals
- 72/100
Model compatibility evidence
- claudeinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- chatgptinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- geminiinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- copilotinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- llamainferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- perplexityinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- mistralinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- grokinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.
What this skill does
# NeMo Evaluator Skill
## Quick Reference
### nemo-evaluator-launcher CLI
```bash # Run evaluation uv run nemo-evaluator-launcher run --config <path.yaml> uv run nemo-evaluator-launcher run --config <path.yaml> -t <a_single_task_to_be_run_by_name> uv run nemo-evaluator-launcher run --config <path.yaml> -t <task_name_1> -t <task_name_2> ... uv run nemo-evaluator-launcher run --config <path.yaml> -o evaluation.nemo_evaluator_config.config.params.limit_samples=10 ...
# Preview the resolved config and the sbatch script without running the evaluation uv run nemo-evaluator-launcher run --config <path.yaml> --dry-run
# Check status (--json for machine-readable output) uv run nemo-evaluator-launcher status <invocation_id> --json
# Get evaluation run info (output paths, slurm job IDs, cluster hostname, etc.) uv run nemo-evaluator-launcher info <invocation_id>
# Copy just the logs (quick — good for debugging) uv run nemo-evaluator-launcher info <invocation_id> --copy-logs ./evaluation-results/
# For artifacts: use `nel info` to discover paths. If remote, SSH to explore and rsync what you need. # If local, just read directly from the paths shown by `nel info`. # ssh <user>@<hostname> "ls <artifacts_path>/" # rsync -avzP <user>@<hostname>:<artifacts_path>/{results.yml,eval_factory_metrics.json,config.yml} ./evaluation-results/<invocation_id>.<job_index>/artifacts/
# Resume a failed/interrupted run (re-sbatches existing run.sub in the original run directory) uv run nemo-evaluator-launcher resume <invocation_id>
# List past runs uv run nemo-evaluator-launcher ls runs --since 1d
# List available evaluation tasks (by default, only shows tasks from the latest released containers) uv run nemo-evaluator-launcher ls tasks uv run nemo-evaluator-launcher ls tasks --from_container nvcr.io/nvidia/eval-factory/simple-evals:26.03 ```
## Workflow
The complete evaluation workflow is divided into the following steps you should follow IN ORDER.
1. Create or modify a config using the `nel-assistant` skill. If the user provides a past run, use its `config.yml` artifact as a starting point. 2. Run the evaluation. See `references/run-evaluation.md` when executing this step. 3. **Monitor progress (MANDATORY after every `nel run`)**: poll status repeatedly until SUCCESS/FAILED. See `references/check-progress.md`. 4. Post-run actions (when terminal state reached): 1. When the evaluation status is `SUCCESS`, analyze the results. See `references/analyze-results.md` when executing this step. 2. When the evaluation status is `FAILED`, debug the failed run. See `references/debug-failed-runs.md` when executing this step.
# Key Facts
- Benchmark-specific info learned during launching/analyzing evals should be added to `references/benchmarks/` - **PPP** = Slurm account / project portfolio code (the `account` field in cluster_config.yaml). When the user says "change PPP to X", update the account value (e.g., `<old_account>` → `<new_account>`). - **Slurm job pairs**: NEL (nemo-evaluator-launcher) submits paired Slurm jobs — a RUNNING job + a PENDING restart job (for when the 4h walltime expires). Never cancel the pending restart jobs — they are expected and necessary. - **HF cache requirement**: For configs with `HF_HUB_OFFLINE=1`, models must be pre-downloaded to the HF cache on each cluster before launching. **Before running a model on a new cluster, always ask the user if the model is already cached there.** If not, on the cluster login node: `python3 -m venv hf_cli && source hf_cli/bin/activate && pip install huggingface_hub` then `HF_HOME=<your_hf_cache_path> hf download <model>` (on lustre-style HPC clusters this is typically under `/lustre/.../<group>/users/<username>/cache/huggingface`). Without this, vLLM will fail with `LocalEntryNotFoundError`. - **`data_parallel_size` is per node**: `dp_size=1` with `num_nodes=8` means 8 model instances total (one per node), load-balanced by haproxy. Do NOT interpret `dp_size` as the global replica count. - **`payload_modifier` interceptor**: The `params_to_remove` list (e.g. `[max_tokens, max_completion_tokens]`) strips those fields from the outgoing payload, intentionally lifting output length limits so reasoning models can think as long as they need. - **Auto-export git workaround**: The export container (`python:3.12-slim`) lacks `git`. When installing the launcher from a git URL, set `auto_export.launcher_install_cmd` to install git first (e.g., `apt-get update -qq && apt-get install -qq -y git && pip install "nemo-evaluator-launcher[all] @ git+...#subdirectory=packages/nemo-evaluator-launcher"`). - **Do NOT use `nemo-evaluator-launcher export --dest local`** — it only writes a summary JSON (`processed_results.json`), it does NOT copy actual logs or artifacts despite accepting `--copy_logs` and `--copy-artifacts` flags. `nel info --copy-artifacts` works but copies everything (very slow for large benchmarks). Preferred approach: use `nel info` to discover paths — if local, read directly; if remote, SSH to explore and rsync only what you need. Note that `nel info` prints standard artifacts but benchmarks produce additional artifacts in subdirs — explore to find them.
Intended uses
- Create or modify a config using the nel-assistant skill. If the user provides a past run, use its config.yml artifact as a starting point.
- Run the evaluation. See references/run-evaluation.md when executing this step.
- Monitor progress (MANDATORY after every nel run): poll status repeatedly until SUCCESS/FAILED. See references/check-progress.md.
- Post-run actions (when terminal state reached):
- When the evaluation status is SUCCESS, analyze the results. See references/analyze-results.md when executing this step.
- When the evaluation status is FAILED, debug the failed run. See references/debug-failed-runs.md when executing this step.
Implementation notes
- Review the source instructions and adapt inputs before running the workflow.
Try it, then tell us how it went
A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.