Skip to main content
Agent skill

Modelopt Run Agentx

Run the AgentX agentic serving benchmark with the SemiAnalysis harness. Use for AgentX setup, dataset downloads, concurrency sweeps, or serving latency and throughput comparisons.

All install options
Quick install · Personal (~/.claude/skills)Project / Cursor / Desktop →
mkdir -p ~/.claude/skills/modelopt-run-agentx && curl -fsSL 'https://raw.githubusercontent.com/NVIDIA/Model-Optimizer/main/skills/modelopt-run-agentx/SKILL.md' -o ~/.claude/skills/modelopt-run-agentx/SKILL.md

Get the complete skill folder, including its scripts and reference files.

When to use this skill

Use Modelopt Run Agentx when an AI agent needs a reusable SKILL.md workflow for this job: Run the AgentX agentic serving benchmark with the SemiAnalysis harness. Use for AgentX setup, dataset downloads, concurrency sweeps, or serving latency and throughput comparisons.

When not to use it

Skip Modelopt Run Agentx when the task is outside the coding category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.

How to install

Where will you use it?

Install scope
One-line install (Claude Code)
mkdir -p ~/.claude/skills/modelopt-run-agentx && curl -fsSL 'https://raw.githubusercontent.com/NVIDIA/Model-Optimizer/main/skills/modelopt-run-agentx/SKILL.md' -o ~/.claude/skills/modelopt-run-agentx/SKILL.md
  1. Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/modelopt-run-agentx/SKILL.md. For a shared project, commit the folder at .claude/skills/modelopt-run-agentx/ instead.
  2. Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
  3. For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.

Full install guide for Claude, Cursor, and Codex →

Try a starter prompt

After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.

Starter prompt
Use the Modelopt Run Agentx skill for this task: Use Modelopt Run Agentx when this documented workflow matches the task. Confirm any required inputs or files before making changes.

What supports this recommendation?

71/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.

Documentation
77/100
Practical detail
68/100
Source evidence
63/100
Source signals
72/100
Model compatibility evidence
  • claudeuntested

    No model-specific signal or recorded compatibility test was found.

  • chatgptuntested

    No model-specific signal or recorded compatibility test was found.

  • geminiuntested

    No model-specific signal or recorded compatibility test was found.

  • copilotuntested

    No model-specific signal or recorded compatibility test was found.

  • llamauntested

    No model-specific signal or recorded compatibility test was found.

  • perplexityuntested

    No model-specific signal or recorded compatibility test was found.

  • mistraluntested

    No model-specific signal or recorded compatibility test was found.

  • grokuntested

    No model-specific signal or recorded compatibility test was found.

Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.

What this skill does

# Run AgentX

## Setup

Resolve the endpoint URL, served model name, matching tokenizer, and requested concurrency from the task. Reuse the model checkpoint. If serving is needed, follow [deployment](../modelopt-deployment/SKILL.md) to download missing weights and launch it. Enable prefix caching, streaming usage, and cached-token reporting. For vLLM, use `--enable-prefix-caching --enable-prompt-tokens-details`. Verify the context limit fits the traces with the model's tokenizer; never silently truncate requests.

Start each benchmark campaign with the latest harness and compatible serving-stack releases unless the user requests specific versions. Record and pin the resolved harness and other code revisions, dataset/model/tokenizer revisions, serving framework (e.g. vLLM), and client/server dependencies or container digests in the run artifacts. Keep them fixed across sweep points, cache formats, and repeats unless they are the variable under test. Record local patches; treat upgrades as a new comparison baseline.

Install the latest [AgentX harness](https://github.com/SemiAnalysisAI/agentx-harness) in a separate client environment. Set `AGENTX_PYTHON` to a Python executable satisfying the latest release's `Requires-Python` metadata on [PyPI](https://pypi.org/project/agentx-harness/). Installing upstream `aiperf` alone is insufficient.

```bash set -euo pipefail export AGENTX_WORKDIR="${AGENTX_WORKDIR:-$PWD/agentx-work}" "${AGENTX_PYTHON:?}" -m venv "$AGENTX_WORKDIR/venv" source "$AGENTX_WORKDIR/venv/bin/activate" python -m pip install --upgrade agentx-harness ```

Set `AGENTX_DATASET` to an accepted with-subagents alias from the [AgentX tutorial](https://github.com/SemiAnalysisAI/agentx-harness/blob/master/docs/tutorials/agentx-mvp.md). Check the current tutorial for supported CLI options and scenario defaults. The public loader downloads and caches the traces and prompt reconstruction assets. An alias, even a dated one, does not select an immutable Hugging Face revision. Resolve the alias's repository to a commit SHA and record both with every run. Before benchmarking, verify that the loader passes that SHA as `revision` to Hugging Face `load_dataset`. Without a revision flag, set the selected Weka loader's `hf_revision` to that SHA in the campaign's client installation.

Prefer the deployed checkpoint's local tokenizer. If it requires custom code, add `--tokenizer-trust-remote-code` only after the user explicitly trusts its repository.

## Choose concurrency

Sweep concurrency upward until KV-cache usage nears capacity, then refine around that point. Resweep when the model, hardware, or cache budget changes. Publish the full sweep and repeat promising points before claiming an advantage.

## Run

In each shell, set `AGENTX_WORKDIR` to the setup directory and set `AGENTX_DATASET`, `AGENTX_URL`, `AGENTX_MODEL`, `AGENTX_TOKENIZER`, `AGENTX_MAX_CONTEXT_LENGTH`, and `AGENTX_CONCURRENCY` from the deployment and sweep. Use the server's base URL and actual context limit. Fix `AGENTX_SEED` across comparisons. First check endpoint health and repeat a long prompt to verify nonzero cached-token usage. Keep the scenario's default trajectory window and duration. Run each concurrency separately with a fresh artifact directory:

```bash set -euo pipefail source "${AGENTX_WORKDIR:?}/venv/bin/activate" export HF_HOME="${HF_HOME:-$AGENTX_WORKDIR/hf-cache}" export AIPERF_DATASET_MMAP_CACHE_DIR="$AGENTX_WORKDIR/dataset-cache" mkdir -p "$AGENTX_WORKDIR/results" AGENTX_RUN_DIR=$(mktemp -d "$AGENTX_WORKDIR/results/run-XXXXXX") python -m pip freeze > "$AGENTX_RUN_DIR/client-packages.txt" aiperf profile \ --scenario inferencex-agentx-mvp \ --url "${AGENTX_URL:?}" --model "${AGENTX_MODEL:?}" \ --tokenizer "${AGENTX_TOKENIZER:?}" --endpoint-type chat \ --public-dataset "${AGENTX_DATASET:?}" \ --max-context-length "${AGENTX_MAX_CONTEXT_LENGTH:?}" \ --concurrency "${AGENTX_CONCURRENCY:?}" \ --random-seed "${AGENTX_SEED:?}" \ --streaming --use-server-token-count --extra-inputs ignore_eos:true \ --artifact-dir "$AGENTX_RUN_DIR" --ui simple \ 2>&1 | tee "$AGENTX_RUN_DIR/client.log" ```

## Compare and report

Use the [benchmarking guide](../modelopt-deployment/references/benchmarking.md#3-run-a-sweep) for sweep isolation, performance metrics, and comparison controls. For AgentX:

- Concurrency counts session trees. Report actual overlapping HTTP requests separately because child sessions can overlap. - Hold KV-cache bytes fixed when comparing KV formats. Keep the dataset, seed, context filter, and warmup fixed, and start a fresh server per point. - Report token-weighted cache hits from server-reported usage, cache usage, and preemptions. Missing usage means cache hits are unknown. - Report `metadata.submission_valid` and any `metadata.submission_invalid_reasons` from the result export. Flag invalid or missing validity instead of treating the run as a valid comparison. - Include request errors, unfinished requests, and warmup failures. Synthetic prompts measure serving performance, not model quality; shortened smoke runs are not benchmark results.

Intended uses

  • Use Modelopt Run Agentx when this documented workflow matches the task.

Implementation notes

  • Review the source instructions and adapt inputs before running the workflow.

Try it, then tell us how it went

A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.