Cli Eval
Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.
Get the complete skill folder, including its scripts and reference files.
When to use this skill
Use Cli Eval when an AI agent needs a reusable SKILL.md workflow for this job: Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.
When not to use it
Skip Cli Eval when the task is outside the coding category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.
How to install
Where will you use it?
mkdir -p ~/.claude/skills/cli-eval # Save SKILL.md to ~/.claude/skills/cli-eval/SKILL.md
- Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/cli-eval/SKILL.md. For a shared project, commit the folder at .claude/skills/cli-eval/ instead.
- Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
- For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.
Full install guide for Claude, Cursor, and Codex →
Try a starter prompt
After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.
Use the Cli Eval skill for this task: Use Cli Eval when this documented workflow matches the task. Confirm any required inputs or files before making changes.
What supports this recommendation?
70/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.
- Documentation
- 77/100
- Practical detail
- 68/100
- Source evidence
- 63/100
- Source signals
- 70/100
Model compatibility evidence
- claudeinferred
The skill text mentions Claude or a closely associated term. This is keyword-derived compatibility; no hands-on model test is recorded.
- chatgptinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- geminiinferred
The skill text mentions Gemini or a closely associated term. This is keyword-derived compatibility; no hands-on model test is recorded.
- copilotinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- llamainferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- perplexityinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- mistralinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- grokinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.
What this skill does
<!-- generated by src/lib/agentSkills/generator.ts; manual edits will be overwritten -->
## Overview
Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.
## Quick install
```bash npm install -g omniroute # or: npx omniroute omniroute --version ```
## Subcommands
### `eval`
**Example:**
```bash omniroute eval ```
### `eval suites`
**Example:**
```bash omniroute eval suites ```
### `eval list`
**Example:**
```bash omniroute eval list ```
### `eval get <suiteId>`
**Example:**
```bash omniroute eval get <suiteId> ```
### `eval create`
**Flags:**
- `--file <path>`
**Example:**
```bash omniroute eval create ```
### `eval run <suiteId>`
**Flags:**
- `-m, --model <id>` - `--combo <name>` - `--concurrency <n>` - `--tag <tag>` - `--watch`
**Example:**
```bash omniroute eval run <suiteId> ```
### `eval list`
**Flags:**
- `--suite <id>` - `--status <s>` - `--since <ts>` - `--limit <n>`
**Example:**
```bash omniroute eval list ```
### `eval get <runId>`
**Example:**
```bash omniroute eval get <runId> ```
### `eval results <runId>`
**Flags:**
- `--failed`
**Example:**
```bash omniroute eval results <runId> ```
### `eval cancel <runId>`
**Flags:**
- `--yes`
**Example:**
```bash omniroute eval cancel <runId> ```
### `eval scorecard <runId>`
**Example:**
```bash omniroute eval scorecard <runId> ```
### `simulate [prompt]`
**Flags:**
- `--file <path>` - `-m, --model <id>` - `--combo <name>` - `--reasoning-effort <level>` - `--thinking-budget <n>` - `--explain`
**Example:**
```bash omniroute simulate [prompt] ```
<!-- skill:custom-start --> <!-- Migrated from skills/omniroute-cli-eval/SKILL.md (preserved curated content) -->
# OmniRoute — CLI Evals
Requires the `omniroute` CLI. See [CLI entry-point skill](https://raw.githubusercontent.com/cheaper-inference/OmniRoute/main/skills/cli-serve/SKILL.md) for install + global flags.
## What are evals?
Evals are automated test suites that score LLM outputs against expected answers or rubrics. OmniRoute stores suites and run results in its local database.
## Eval suites
```bash omniroute eval suites list # List all eval suites omniroute eval suites list --json # JSON output
omniroute eval suites get <suiteId> # Full suite definition ```
### Create a suite
```bash omniroute eval suites create \ --name "code-quality" \ --rubric "exact-match" \ --samples-file ./samples.jsonl # JSONL: {input, expected_output} ```
Rubric options: `exact-match`, `contains`, `llm-judge`, `regex`.
`--samples-file` format (one JSON object per line):
```jsonl {"input": "What is 2+2?", "expected_output": "4"} {"input": "Translate 'hello' to Spanish", "expected_output": "hola"} ```
## Run an eval
```bash omniroute eval suites run <suiteId> \ --model claude-sonnet-4-6 # Run suite against a specific model
omniroute eval suites run <suiteId> \ --model gpt-4o \ --watch # Live TUI progress (EvalWatch) ```
The run is asynchronous. Use `--watch` for a live terminal dashboard or poll manually:
```bash RUN_ID=$(omniroute eval suites run <suiteId> --model claude-sonnet-4-6 --output json | jq -r '.id') omniroute eval get $RUN_ID ```
## Manage runs
```bash omniroute eval list # List all eval runs omniroute eval list --json
omniroute eval get <runId> # Run details (status, model, score) omniroute eval results <runId> # Per-sample results omniroute eval scorecard <runId> # Full scorecard with pass/fail per sample omniroute eval cancel <runId> # Cancel a running eval ```
## Scorecard output
```bash omniroute eval scorecard <runId> --output json ```
Response fields per sample:
```json { "id": "sample-1", "score": 0.95, "passed": true, "input": "What is 2+2?", "output": "4", "expected": "4" } ```
## Comparing models
Run the same suite against multiple models and compare:
```bash for MODEL in claude-sonnet-4-6 gpt-4o gemini-2.0-flash; do omniroute eval suites run $SUITE_ID --model $MODEL --output json | jq '{model: .model, score: .score}' done ```
## CI integration
```bash # Run and fail CI if score drops below threshold SCORE=$(omniroute eval suites run $SUITE_ID --model claude-sonnet-4-6 --output json | jq -r '.score') python3 -c "import sys; score=float('$SCORE'); sys.exit(0 if score >= 0.90 else 1)" ```
## Errors
- `suites create` fails with `invalid rubric` → use one of: `exact-match`, `contains`, `llm-judge`, `regex` - `suites run` returns `model not found` → verify model ID with `omniroute models --search <name>` - `eval get` shows `status: failed` → check `omniroute logs --search eval` for error details - `scorecard` returns empty results → the run may still be `running`; poll `omniroute eval get <runId>` until `status` is `completed` <!-- skill:custom-end -->
Intended uses
- Use Cli Eval when this documented workflow matches the task.
Implementation notes
- Review the source instructions and adapt inputs before running the workflow.
Try it, then tell us how it went
A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.