Vllm
Deploy and serve LLMs with vLLM behind an OpenAI-compatible endpoint, with tool calling enabled for agent workloads.
Get the complete skill folder, including its scripts and reference files.
When to use this skill
Use Vllm when an AI agent needs a reusable SKILL.md workflow for this job: Deploy and serve LLMs with vLLM behind an OpenAI-compatible endpoint, with tool calling enabled for agent workloads.
When not to use it
Skip Vllm when the task is outside the coding category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.
How to install
Where will you use it?
mkdir -p ~/.claude/skills/vllm # Save SKILL.md to ~/.claude/skills/vllm/SKILL.md
- Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/vllm/SKILL.md. For a shared project, commit the folder at .claude/skills/vllm/ instead.
- Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
- For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.
Full install guide for Claude, Cursor, and Codex →
Try a starter prompt
After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.
Use the Vllm skill for this task: Use Vllm when this documented workflow matches the task. Confirm any required inputs or files before making changes.
What supports this recommendation?
69/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.
- Documentation
- 72/100
- Practical detail
- 68/100
- Source evidence
- 63/100
- Source signals
- 70/100
Model compatibility evidence
- claudeuntested
No model-specific signal or recorded compatibility test was found.
- chatgptinferred
The skill text mentions ChatGPT or a closely associated term. This is keyword-derived compatibility; no hands-on model test is recorded.
- geminiuntested
No model-specific signal or recorded compatibility test was found.
- copilotuntested
No model-specific signal or recorded compatibility test was found.
- llamainferred
The skill text mentions Llama or a closely associated term. This is keyword-derived compatibility; no hands-on model test is recorded.
- perplexityuntested
No model-specific signal or recorded compatibility test was found.
- mistraluntested
No model-specific signal or recorded compatibility test was found.
- grokuntested
No model-specific signal or recorded compatibility test was found.
Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.
What this skill does
# vLLM Serving
vLLM serves open-weight LLMs on local GPUs with high-throughput inference behind an OpenAI-compatible API, ready for chat and agent workloads.
## Before you start
If the user's message only invokes this skill (e.g. "use vllm skill") without a concrete request, ask the user what they want. Do not run any command until the goal is clear.
Ask the user which model to serve; if they have no preference, recommend the small default [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B). Also ask what context length the workload needs.
vLLM needs an NVIDIA or AMD GPU. Engine choice follows the user's preference: Ollama also runs on GPUs and is the simpler default — pick vLLM for high-throughput serving, and Ollama on macOS or CPU-only machines, which vLLM does not serve. Confirm the hardware first:
```bash nvidia-smi # NVIDIA: GPU model and free VRAM (AMD ROCm: rocm-smi) python3 --version # a recent Python is required ```
The model must fit the available VRAM — model size and context length drive the serve flags below.
## Suggested workflow
1. Ask the user which model to serve; with no preference, recommend [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B). 2. Pick the engine the user prefers: vLLM for high-throughput GPU serving; Ollama is the simple default and the choice on macOS or CPU-only machines. 3. Serve on a free port, with the tool-calling flags whenever agents will call it (see below). 4. Verify with `curl http://localhost:8000/v1/models`. 5. Register the endpoint: `penguin config model add ... --client-type openai-chat --base-url http://localhost:8000/v1` — a served model is not visible to Penguin until added. 6. Confirm the new entry with `penguin config model list`.
## Install
Use a fresh virtual environment (or `uv`):
```bash python3 -m venv .venv && source .venv/bin/activate pip install vllm ```
## Serve
```bash vllm serve Qwen/Qwen3.5-0.8B --port 8000 ```
This exposes an OpenAI-compatible API at `http://localhost:8000/v1`. Key flags:
- `--served-model-name <name>` — the model id clients request (defaults to the model path). - `--api-key <key>` — require this bearer token on every request. - `--max-model-len <n>` — context window; agent sessions need a large one. - `--gpu-memory-utilization <0..1>` — fraction of VRAM to claim (default 0.9). - `--tensor-parallel-size <n>` — shard across `n` GPUs. - `--dtype <auto|bfloat16|float16>` and `--quantization <awq|gptq|fp8>` — precision and quantized weights.
If the port is taken, pick a free one — never kill a process already listening on it.
## Tool calling — required for agents
Agent harnesses (PenguinHarness included) send `tools` with their requests. vLLM must opt in at startup:
```bash vllm serve Qwen/Qwen3.5-0.8B --enable-auto-tool-choice --tool-call-parser hermes ```
Choose the parser for the model family — e.g. `hermes` for Qwen models, `llama3_json` for Llama models. Without these flags, requests that set tool_choice fail with `400 "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set`.
## Verify
```bash curl http://localhost:8000/v1/models ```
## Register with PenguinHarness
Model configuration is the penguin CLI's job — `penguin config model add` registers an endpoint and `penguin config model list` shows what has been registered. A served model is not visible to Penguin until you add it:
```bash penguin config model add --provider custom --client-type openai-chat \ --base-url http://localhost:8000/v1 --model-id <served-model-name> --api-key <key> penguin config model list # the new entry should now be listed ```
## Troubleshooting
- Out of memory at startup: lower `--gpu-memory-utilization` or `--max-model-len`, or serve a quantized model. - Long prompts truncated or context-length errors: raise `--max-model-len` (bounded by VRAM). - `400` on tool calls: restart the server with the tool-calling flags above.
Intended uses
- Use Vllm when this documented workflow matches the task.
Implementation notes
- Review the source instructions and adapt inputs before running the workflow.
Try it, then tell us how it went
A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.