Modelopt Monitor
Monitor submitted jobs (PTQ, evaluation, deployment) on SLURM clusters. Use when the user asks "check job status", "is my job done", "monitor my evaluation", "what's the status of the PTQ", "check on job" followed by a SLURM job ID, or after any skill submits a long-running job. Also triggers on "nel status", "squeue", or any request to check progress of a previously submitted job.
mkdir -p ~/.claude/skills/modelopt-monitor && curl -fsSL 'https://raw.githubusercontent.com/NVIDIA/Model-Optimizer/main/skills/modelopt-monitor/SKILL.md' -o ~/.claude/skills/modelopt-monitor/SKILL.md
Get the complete skill folder, including its scripts and reference files.
When to use this skill
Use Modelopt Monitor when an AI agent needs a reusable SKILL.md workflow for this job: Monitor submitted jobs (PTQ, evaluation, deployment) on SLURM clusters. Use when the user asks "check job status", "is my job done", "monitor my evaluation", "what's the status of the PTQ", "check on job" followed by a SLURM job ID, or after any skill submits a long-running job. Also triggers on "nel status", "squeue", or any request to check progress of a previously submitted job.
When not to use it
Skip Modelopt Monitor when the task is outside the coding category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.
How to install
Where will you use it?
mkdir -p ~/.claude/skills/modelopt-monitor && curl -fsSL 'https://raw.githubusercontent.com/NVIDIA/Model-Optimizer/main/skills/modelopt-monitor/SKILL.md' -o ~/.claude/skills/modelopt-monitor/SKILL.md
- Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/modelopt-monitor/SKILL.md. For a shared project, commit the folder at .claude/skills/modelopt-monitor/ instead.
- Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
- For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.
Full install guide for Claude, Cursor, and Codex →
Try a starter prompt
After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.
Use the Modelopt Monitor skill for this task: Auto-monitor — another skill (PTQ, evaluation, deployment) just submitted a job. Register the job and set up monitoring immediately. Confirm any required inputs or files before making changes.
What supports this recommendation?
76/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.
- Documentation
- 82/100
- Practical detail
- 80/100
- Source evidence
- 63/100
- Source signals
- 72/100
Model compatibility evidence
- claudeinferred
The skill text mentions Claude or a closely associated term. This is keyword-derived compatibility; no hands-on model test is recorded.
- chatgptuntested
No model-specific signal or recorded compatibility test was found.
- geminiuntested
No model-specific signal or recorded compatibility test was found.
- copilotuntested
No model-specific signal or recorded compatibility test was found.
- llamauntested
No model-specific signal or recorded compatibility test was found.
- perplexityuntested
No model-specific signal or recorded compatibility test was found.
- mistraluntested
No model-specific signal or recorded compatibility test was found.
- grokuntested
No model-specific signal or recorded compatibility test was found.
Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.
What this skill does
# Job Monitor
Monitor jobs submitted to SLURM clusters — PTQ quantization, NEL evaluation, model deployment, or raw SLURM jobs.
## When to use
1. **Auto-monitor** — another skill (PTQ, evaluation, deployment) just submitted a job. Register the job and set up monitoring immediately. 2. **User-initiated** — user asks about a job status. Check the current session registry first; if the job is not registered there, use the discovery steps below.
---
## Job Registry
Active jobs are tracked in per-session registries under `.claude/agents/`. This avoids multiple agents clobbering one shared registry when they run at the same time.
Use the current agent session id as `<session_id>`:
- Claude Code: `$CLAUDE_CODE_SESSION_ID`, or the `session_id` field from hook input - Codex: `$CODEX_THREAD_ID` - If no session id is available, create a stable id for the current terminal session and reuse it for every job registered by that agent
Registry layout:
```text .claude/agents/ <session_id>/ active_jobs.json ```
Each session's `active_jobs.json` is a JSON array:
```json [ { "type": "nel", "id": "<invocation_id or slurm_job_id>", "host": "<cluster_hostname>", "user": "<ssh_user>", "submitted": "YYYY-MM-DD HH:MM", "description": "<what this job does>", "last_status": "<last known status>", "owner": { "agent": "claude-code|codex|manual", "session_id": "<session_id>" } } ] ```
`type` is one of: `nel`, `slurm`, `launcher`.
---
## On Job Submission
Every time a job is submitted (by any skill or manually):
1. **Add an entry** to `.claude/agents/<session_id>/active_jobs.json`. Create the session directory and file if they don't exist. 2. **Start a durable monitor** (if one isn't already watching the registry) that polls this session's registered jobs until they reach terminal status. Prefer the Claude Code `Monitor` tool when it is available: write a small watcher that reads `.claude/agents/<session_id>/active_jobs.json`, checks every job with the appropriate method below, prints state-change events, updates `last_status`, removes terminal jobs from the session registry, and exits when no active jobs remain for this session.
The monitor should terminate naturally when every registered job has reached a terminal state. If the `Monitor` tool is not available in the current harness, run an equivalent background process that implements the same loop and lets the agent resume/restart when the process exits.
Always do both steps. Don't try to predict job duration.
---
## On Monitor Event / Status Check
Whether triggered by monitor output or by the user asking "check status":
1. **Read the registry** from `.claude/agents/<session_id>/active_jobs.json` 2. **Check each job** using the appropriate method (see below) 3. **Report only state changes** — compare against `last_status` in registry 4. **Update `last_status`** in the session registry 5. **Remove completed jobs** — any job in a terminal state (COMPLETED, FAILED, CANCELLED, KILLED, TIMEOUT, NODE_FAIL, OUT_OF_MEMORY, PREEMPTED, BOOT_FAIL, DEADLINE) 6. **If no active jobs remain** — let the monitor exit
---
## How to Check Each Job Type
Each check method has its **own** status vocabulary. A watcher that mixes them (e.g. uses SLURM's `COMPLETED` terminal-state regex against `nel status` output) will silently never fire terminal transitions. Always match against the vocabulary of the source you're polling.
### NEL jobs (`type: nel`)
- **Check:** `nel status <id>`.
```bash extract_nel_state() { local jid="$1" nel_bin="${NEL:-nel}" output state_col output=$("$nel_bin" status "$jid" 2>&1) state_col=$(echo "$output" \ | awk -F'|' -v prefix="$jid." 'index($1, prefix) == 1 { print $2; exit }') [ -z "$state_col" ] && state_col="$output" echo "$state_col" \ | LC_ALL=C tr '[:lower:]' '[:upper:]' \ | awk 'match($0, /(PENDING|RUNNING|SUCCESS|FAILED|KILLED|ERROR|NOT[[:space:]]+FOUND)/) { print substr($0, RSTART, RLENGTH); exit }' \ | sed 's/[[:space:]][[:space:]]*/ /g' }
is_nel_terminal() { case "$(extract_nel_state "$1")" in SUCCESS|FAILED|KILLED|ERROR|"NOT FOUND") return 0 ;; *) return 1 ;; esac } ```
- **On completion:** `nel info <id>` to fetch results. - **On failure:** `nel info <id> --logs` then inspect server/client/SLURM logs via SSH.
### Launcher jobs (`type: launcher`)
- **Check:** Tail the launcher's background output file for key events. - **Key events:** experiment ID, SLURM job ID, container import, calibration progress, export path, final status. - **On failure:** Look for `Traceback`, `Error`, or `FAILED` in the output.
### Raw SLURM jobs (`type: slurm`)
- **Check:** `sacct`; use `sacct` for the termination check because `squeue` can lag in `COMPLETING` after `sacct` reports a terminal state.
```bash extract_slurm_state() { local jid="$1" host="$2" ssh "$host" "sacct -j $jid -X --format=State --noheader -P 2>/dev/null | head -1" \ | sed 's/^[[:space:]]*//;s/[[:space:]]*$//' \ | sed 's/^CANCELLED by .*/CANCELLED/' }
is_slurm_terminal() { case "$(extract_slurm_state "$1" "$2")" in COMPLETED|FAILED|CANCELLED|TIMEOUT|NODE_FAIL|OUT_OF_MEMORY|PREEMPTED|BOOT_FAIL|DEADLINE) return 0 ;; *) return 1 ;; esac } ```
- **On completion:** `ssh <host> "sacct -j <id> --format=State,ExitCode,Elapsed -n"`. - **On failure:** Check the job's output log file.
---
## Identifying Jobs (user-initiated, no ID given)
When the user asks about a job without specifying an ID, check in order:
1. `.claude/agents/<current_session_id>/active_jobs.json` — current agent's jobs 2. `nel ls runs --since 1d` — recent NEL runs 3. `ssh <host> "squeue -u <user>"` — active SLURM jobs 4. `ls -lt tools/launcher/experiments/cicd/ | head -10` — recent launcher experiments
---
## Reporting Guidelines
- **Report state changes proactively** — PENDING → RUNNING, or job completes - **Aggregate multiple jobs** — "2 of 4 completed (MMLU-Pro: 42.3%, GSM8K: 67.1%), 1 running, 1 pending" - **Summarize, don't echo** — interpret events ("Calibration complete, exporting checkpoint") not raw logs - **On failure, diagnose immediately** — check logs and report root cause without waiting for user to ask - **Minimize noise** — don't report "still running" unless the user is actively asking
Intended uses
- Auto-monitor — another skill (PTQ, evaluation, deployment) just submitted a job. Register the job and set up monitoring immediately.
- User-initiated — user asks about a job status. Check the current session registry first; if the job is not registered there, use the discovery steps below.
Implementation notes
- Review the source instructions and adapt inputs before running the workflow.
Try it, then tell us how it went
A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.