Skip to main content
Agent skill

Modelopt Benchmark Model Kernels

Inspect Hugging Face decoder layers on meta tensors and plan or run per-rank BF16, FP8, and NVFP4 GEMM or fused-MoE microbenchmarks with the bundled scripts and a local FlashInfer checkout. Use when choosing a model, GPU, TP, EP, or M/token-concurrency sweep; deriving common fused QKV and gate/up shapes without loading checkpoint weights; or using FlashInfer benchmark utilities. Do not use for end-to-end server throughput or request latency.

All install options
Quick install · Personal (~/.claude/skills)Project / Cursor / Desktop →
mkdir -p ~/.claude/skills/modelopt-benchmark-model-kernels && curl -fsSL 'https://raw.githubusercontent.com/NVIDIA/Model-Optimizer/main/skills/modelopt-benchmark-model-kernels/SKILL.md' -o ~/.claude/skills/modelopt-benchmark-model-kernels/SKILL.md

Get the complete skill folder, including its scripts and reference files.

When to use this skill

Use Modelopt Benchmark Model Kernels when an AI agent needs a reusable SKILL.md workflow for this job: Inspect Hugging Face decoder layers on meta tensors and plan or run per-rank BF16, FP8, and NVFP4 GEMM or fused-MoE microbenchmarks with the bundled scripts and a local FlashInfer checkout. Use when choosing a model, GPU, TP, EP, or M/token-concurrency sweep; deriving common fused QKV and gate/up shapes without loading checkpoint weights; or using FlashInfer benchmark utilities. Do not use for end-to-end server throughput or request latency.

When not to use it

Skip Modelopt Benchmark Model Kernels when the task is outside the coding category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.

How to install

Where will you use it?

Install scope
One-line install (Claude Code)
mkdir -p ~/.claude/skills/modelopt-benchmark-model-kernels && curl -fsSL 'https://raw.githubusercontent.com/NVIDIA/Model-Optimizer/main/skills/modelopt-benchmark-model-kernels/SKILL.md' -o ~/.claude/skills/modelopt-benchmark-model-kernels/SKILL.md
  1. Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/modelopt-benchmark-model-kernels/SKILL.md. For a shared project, commit the folder at .claude/skills/modelopt-benchmark-model-kernels/ instead.
  2. Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
  3. For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.

Full install guide for Claude, Cursor, and Codex →

Try a starter prompt

After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.

Starter prompt
Use the Modelopt Benchmark Model Kernels skill for this task: Use Modelopt Benchmark Model Kernels when this documented workflow matches the task. Confirm any required inputs or files before making changes.

What supports this recommendation?

71/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.

Documentation
77/100
Practical detail
68/100
Source evidence
63/100
Source signals
72/100
Model compatibility evidence
  • claudeinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • chatgptinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • geminiinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • copilotinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • llamainferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • perplexityinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • mistralinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • grokinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.

What this skill does

# Benchmark Model Kernels

Plan a single-GPU microbenchmark for the per-rank shapes of an intended deployment. The scripts never load checkpoint weights, launch distributed workers, or measure collectives, serving throughput, or request latency.

## Interview, one decision per message

Ask exactly one unresolved decision per message with two or three concrete options, state the default, and wait for the answer. Recommend an option only when it is substantively better. Skip decisions already answered. Follow this order:

1. **Model** — a local directory, a `config.json`, or a Hub ID (equivalent; no default). The script builds the model on meta tensors from configuration only. Do not enable `--trust-remote-code` without explicit approval; pin `--revision <commit>` when it is needed. 2. **TP and EP** — accept both directly, or derive them from the intended deployment's GPU model and count and confirm. Defaults: `TP=1`, `EP=1`. Derive parallelism from the intended deployment, never from the GPU that runs the microbenchmark. The script validates every sharding rule (divisibility, GQA replication, expert partitioning) and errors loudly. 3. **M sweep** — balanced default (recommended): `1 8 64 512`; decode-focused: `1 4 16 32`; throughput-focused: `64 256 1024 4096`. M is roughly the tokens scheduled per step (decode: active sequences), not endpoint concurrency. With EP, an MoE row models one rank's share of the global batch: a global batch of B tokens corresponds to the column M = B/EP, so do not compare different EP values at the same M. 4. **Shape preview** — always run this before anything else; no FlashInfer or GPU needed:

```bash python "$SKILL_DIR/scripts/benchmark_model.py" <model> \ --tp <tp> --ep <ep> --ms <m1> <m2> ... --print_only ```

Review the printed shapes and the MoE tuple with the user. `# unsupported:` lines mean the list is partial and the script exits nonzero — handle those via **Manual supplements** below. 5. **FlashInfer checkout and GPU** — the full benchmark needs a FlashInfer *source checkout* containing `benchmarks/flashinfer_benchmark.py`; the installed wheel alone is not enough. Prefer a clean checkout matching the installed `flashinfer` version; ask before cloning or installing anything. On the benchmark machine, check `nvidia-smi` and package versions, verify the target GPU is idle (concurrent work on the same GPU skews timings), and verify CUPTI timing with a tiny `bench_gpu_time(..., enable_cupti=True)` probe — a warning that falls back to CUDA events is a failure (the `cupti-python`/`nvidia-cuda-cupti` packages must match PyTorch's CUDA major). Ask for a GPU index only when several are visible. Pick a fresh workdir and state it. 6. **Full benchmark**:

```bash CUDA_VISIBLE_DEVICES=<gpu-index> \ python "$SKILL_DIR/scripts/benchmark_model.py" <model> \ --tp <tp> --ep <ep> --ms <m1> <m2> ... \ --flashinfer_repo <flashinfer-repo> --workdir <workdir> ```

Run a short plumbing check first (append `--dry_run_iters 1 --num_iters 3 --no_autotune`, throwaway workdir), especially after changing FlashInfer versions or shape logic; never present it as a performance result. Then run with defaults. The first fused-MoE build or autotune can take several minutes; its cache is reused.

## Rules the scripts own

Do not restate, re-derive, or override these — the code enforces them:

- `benchmark_model.py` derives `--nks` (as `N,K,module-name` triples) and all `--moe_*` arguments; overrides are rejected. - Row labels are logical shapes; the runner applies vLLM's physical padding. Report both shapes whenever they differ. - A failed case writes FlashInfer's error message (or a pointer to `driver.log`) into its `combined_results.csv` cell, and the command exits nonzero after the table is written. Never present a partial table as a successful benchmark; read `driver.log` before rerunning anything.

## Known backend and driver limits

- `mm_fp4` TensorRT-LLM needs `N % 128 == 0` (shuffled weight layout): its cell reports no result row, or on stock 0.6.x drivers an empty assertion that fails every `mm_fp4` backend for that shape. - `mm_fp8` (trtllm_low_latency) needs `K % 128 == 0`. - MoE rows cover FlashInfer's CUTLASS fused MoE, the trtllm-gen NVFP4 and per-tensor FP8 MoE, and the CuteDSL NVFP4 MoE. The CuteDSL MoE row appears only for Swiglu models (the kernel supports nothing else); the `cutedsl` and `trtllm` backends require recent GPUs and report per-case errors elsewhere. - Gated NVFP4 CUTLASS MoE with `2F % 128 != 0` per rank fails — vLLM raises instead of padding — often as a `CUDA error: misaligned address`. Prefer EP over TP for the experts to keep the per-rank width legal. - Do not benchmark a padded shape to dodge these limits and call it vLLM-equivalent; report the limit instead.

## Manual supplements

For each `# unsupported:` layout from the preview: inspect that module's forward path and the intended runtime's TP/EP sharding, derive the per-rank shape (never guess sharding from a weight shape alone), and benchmark only the missing shape:

```bash CUDA_VISIBLE_DEVICES=<gpu-index> \ python "$SKILL_DIR/scripts/benchmark_via_builtin.py" \ --flashinfer_repo <flashinfer-repo> --ms <m1> <m2> ... \ --nks <n>,<k>,<name> --workdir <workdir> ```

Label these rows as manual supplements; this is not a substitute for running `benchmark_model.py` first. Shell-quote user-supplied paths.

## Report

Report the command, GPU, versions, TP/EP, M values, shapes (logical and physical where they differ), warnings, and the artifacts: `testlist.txt`, `driver.log` (full driver output), `builtin_results.csv` (milliseconds, success-only), and `combined_results.csv` (long form, microseconds: columns `module_name, M, N, K, backend, with_quant, runtime` in `GEMM` and `MoE` sections; modules fused into one GEMM are joined with `|` in `module_name`, while distinct same-shape modules appear as duplicated rows sharing one measurement; MoE rows keep the `H= F= E= top_k=` parameter line and leave N/K empty). In quantization-recipe terms: `bf16` rows are the unquantized W16A16 baseline, `fp8` rows are per-tensor W8A8, and `nvfp4` rows are W4A4. Plain quantized rows time the kernel with pre-quantized activations; `*_with_quant` rows add a separately measured activation-quantization time in the scale-factor layout that backend consumes, except the NVFP4 CUTLASS MoE row, which is a single fused measurement. MoE routing is synthetic: uniform expert distribution everywhere (real skewed routing is slower), and the trtllm-gen rows, which route in-kernel, use a fixed `renormalize` method regardless of the model's routing scheme. CUTLASS and CuteDSL MoE rows receive precomputed expert indices and exclude routing-selection cost entirely. No row includes the router GEMM itself. These are kernel times only — never describe them as end-to-end latency or throughput; they omit weights, layer frequency, communication, KV cache, and scheduling.

Intended uses

  • Use Modelopt Benchmark Model Kernels when this documented workflow matches the task.

Implementation notes

  • Review the source instructions and adapt inputs before running the workflow.

Try it, then tell us how it went

A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.