Skip to main content
Agent skill

Agent Tracing

Use for agent traces by operation ID (拉线上 tracing): failed tools, arguments/results, available tools, execution location, LLM calls and context.

All install options
Quick install · Personal (~/.claude/skills)Project / Cursor / Desktop →
mkdir -p ~/.claude/skills/agent-tracing && curl -fsSL 'https://raw.githubusercontent.com/lobehub/lobehub/canary/.agents/skills/agent-tracing/SKILL.md' -o ~/.claude/skills/agent-tracing/SKILL.md

Get the complete skill folder, including its scripts and reference files.

When to use this skill

Use Agent Tracing when an AI agent needs a reusable SKILL.md workflow for this job: Use for agent traces by operation ID (拉线上 tracing): failed tools, arguments/results, available tools, execution location, LLM calls and context.

When not to use it

Skip Agent Tracing when the task is outside the coding category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.

How to install

Where will you use it?

Install scope
One-line install (Claude Code)
mkdir -p ~/.claude/skills/agent-tracing && curl -fsSL 'https://raw.githubusercontent.com/lobehub/lobehub/canary/.agents/skills/agent-tracing/SKILL.md' -o ~/.claude/skills/agent-tracing/SKILL.md
  1. Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/agent-tracing/SKILL.md. For a shared project, commit the folder at .claude/skills/agent-tracing/ instead.
  2. Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
  3. For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.

Full install guide for Claude, Cursor, and Codex →

Try a starter prompt

After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.

Starter prompt
Use the Agent Tracing skill for this task: Use Agent Tracing when this documented workflow matches the task. Confirm any required inputs or files before making changes.

What supports this recommendation?

70/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.

Documentation
75/100
Practical detail
68/100
Source evidence
63/100
Source signals
72/100
Model compatibility evidence
  • claudeinferred

    The skill text mentions Claude or a closely associated term. This is keyword-derived compatibility; no hands-on model test is recorded.

  • chatgptinferred

    The skill text mentions ChatGPT or a closely associated term. This is keyword-derived compatibility; no hands-on model test is recorded.

  • geminiinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • copilotinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • llamainferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • perplexityinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • mistralinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

  • grokinferred

    The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.

Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.

What this skill does

# Agent Tracing CLI Guide

`@lobechat/agent-tracing` is a zero-config local dev tool that records agent execution snapshots to disk and provides a CLI to inspect them.

## How It Works

In `NODE_ENV=development`, `AgentRuntimeService.executeStep()` automatically records each step to `.agent-tracing/` as partial snapshots. When the operation completes, the partial is finalized into a complete `ExecutionSnapshot` JSON file.

**Data flow**: executeStep loop -> build `StepPresentationData` -> write partial snapshot to disk -> on completion, finalize to `.agent-tracing/{timestamp}_{traceId}.json`

**Context engine capture**: In `RuntimeExecutors.ts`, the `call_llm` executor calls `ctx.tracingContextEngine(input, output, metadata)` after `serverMessagesEngine()` processes messages. `AgentRuntimeService.executeStep` buffers the call per step and forwards it to `OperationTraceRecorder.appendStep` as the typed `contextEngine` field. CE flows through this side channel rather than the `events` array so its heavy payload (agentDocuments, systemRole, …) never enters the Redis state pipeline (LOBE-9110). `metadata` carries the pipeline's per-request decision records (trim stats, cache-warmth gates, truncation counts).

## Package Location

``` packages/agent-tracing/ src/ types.ts # ExecutionSnapshot, StepSnapshot, SnapshotSummary store/ types.ts # ISnapshotStore interface file-store.ts # FileSnapshotStore (.agent-tracing/*.json) recorder/ index.ts # appendStepToPartial(), finalizeSnapshot() viewer/ index.ts # Terminal rendering: renderSnapshot, renderStepDetail, renderMessageDetail, renderSummaryTable, renderPayload, renderPayloadTools, renderMemory cli/ index.ts # CLI entry point (#!/usr/bin/env bun) inspect.ts # Inspect command (default) partial.ts # Partial snapshot commands (list, inspect, clean) index.ts # Barrel exports ```

## Data Storage

- Completed snapshots: `.agent-tracing/{ISO-timestamp}_{traceId-short}.json` - Latest symlink: `.agent-tracing/latest.json` - In-progress partials: `.agent-tracing/_partial/{operationId}.json` - Downloaded remote snapshots: `.agent-tracing/_remote/{operationId}.json` - `FileSnapshotStore` resolves from `process.cwd()` — **run CLI from the repo root**

## Remote Traces (Production / Staging)

Server deployments also upload completed snapshots to object storage (zstd-compressed; the key is stored in `agent_operations.trace_s3_key`).

**Preferred: `lh trace op`.** The server resolves the key and signs the object for the caller's own scope, so a LobeHub login is the only requirement — no `TRACING_BASE_URL`, no bucket domain, and no SQL to turn a topic id into an operation id:

```bash lh trace op list --topic tpc_xxx # operations of a topic, newest first, with a TRACE column lh trace op inspect op_xxx_agt_xxx_tpc_xxx_xxxx lh trace op inspect op_xxx_agt_xxx_tpc_xxx_xxxx -T # tool injection (enabledToolIds, manifests) ```

`TRACE = —` in `list` means no snapshot was recorded for that run (it predates trace upload, or upload was off). A recorded snapshot can still 404 in storage after its retention window.

Backend: the `agentTrace` lambda router (`getSnapshotUrl` / `listOperations`). Note it is `blocked` for restricted API keys, so these commands need a real session, not a scoped key.

**Fallback: the standalone `agent-tracing` CLI.** It has no LobeHub session, so it builds the object URL itself and needs the bucket's public domain configured:

- env var: `TRACING_BASE_URL=https://<bucket-public-domain>/agent-traces` - or `.agent-tracing/.env` in the repo root with the same `TRACING_BASE_URL=...` line

The deployment-specific value is private to each deployment and intentionally not recorded in this repo. Find the operation id by hand first:

```sql SELECT id, trace_s3_key FROM agent_operations WHERE topic_id = 'tpc_xxx'; ```

Either way the snapshot is cached to `.agent-tracing/_remote/<opId>.json`, and every `inspect` flag works the same as for local traces.

Implementation: `packages/agent-tracing/src/store/loadSnapshot.ts` (resolution order: local store → `_remote/` cache → injected `resolveDownloadUrl` → `TRACING_BASE_URL`) and `store/remote-store.ts` (URL built as `{base}/{agentId}/{topicId}/{opId}.json.zst`). Reading a compressed snapshot needs Node >= 22.15.

## Goal Trajectories

A goal is one complete _goal_ execution the way an operation is one complete agent execution, so it gets the same trace format one level up: `GoalTrajectory : AdvanceSnapshot` mirrors `ExecutionSnapshot : StepSnapshot`. There is no table of advances, exactly as there is no `agent_steps` table — `goal_traces` holds one rollup row per goal plus the object key, and the detail lives in the object.

The leaves join back down: an advance records the `operationId`s it put in flight (on `tick.effects[].operationId`), so `lh trace op inspect <opId>` continues from where the goal trace stops.

```bash agent-tracing goal # list local goal trajectories agent-tracing goal goal_xxx # the run: triggers, outcomes, graph, gates, ops agent-tracing goal goal_xxx -a 3 # one advance in full, with the frontier it ranked agent-tracing goal goal_xxx -j # raw JSON ```

What each tick records is the **decision input**, not just the result: the graph it read (as a delta against the previous tick), the budget it evaluated, the responsible task's state, and every eligible work node **including the ones it passed over**. Without the losers a trace cannot answer "why not that node".

**Storage** follows the operation switch, so a deployment that keeps one keeps the other:

- Completed: `goal-traces/{goalId}.json.zst` - In progress: `goal-traces/_partial/{goalId}.json.zst` (an unfinished long-horizon goal is the normal thing to inspect) - Dev: `.goal-tracing/{goalId}.json`

**Replay.** `replayGoalAgainstCurrentCoordinator(trajectory)` re-runs the real `decideNextMove` over the recorded inputs and reports `{advanceSeq, tickIndex, field, recorded, replayed}` wherever the current coordinator would now choose differently. This reproduces the coordinator's _decisions_ exactly; it says nothing about whether the dispatched work would have gone the same way.

Implementation: `packages/agent-tracing/src/goal/`, recorded through `apps/server/src/services/goal/advanceGoal.ts` (the single funnel every advance passes through) via the `onDecision` side channel on `GoalService.tick`.

## CLI Commands

All commands run from the **repo root**:

```bash # View latest trace (tree overview, `inspect` is the default command) agent-tracing agent-tracing inspect agent-tracing inspect <traceId> agent-tracing inspect latest

# List recent snapshots agent-tracing list agent-tracing list -l 20

# Inspect specific step (-s is short for --step) agent-tracing inspect <traceId> -s 0

# View messages (-m is short for --messages) agent-tracing inspect <traceId> -s 0 -m

# View full content of a specific message (by index shown in -m output) agent-tracing inspect <traceId> -s 0 --msg 2 agent-tracing inspect <traceId> -s 0 --msg-input 1

# View tool call/result details (-t is short for --tools) agent-tracing inspect <traceId> -s 1 -t

# View raw events (-e is short for --events) agent-tracing inspect <traceId> -s 0 -e

# View runtime context (-c is short for --context) agent-tracing inspect <traceId> -s 0 -c

# View context engine input overview (-p is short for --payload) agent-tracing inspect <traceId> -p agent-tracing inspect <traceId> -s 0 -p

# View available tools in payload (-T is short for --payload-tools) agent-tracing inspect <traceId> -T agent-tracing inspect <traceId> -s 0 -T

# View user memory (-M is short for --memory) agent-tracing inspect <traceId> -M agent-tracing inspect <traceId> -s 0 -M

# Raw JSON output (-j is short for --json) agent-tracing inspect <traceId> -j agent-tracing inspect <traceId> -s 0 -j

# List in-progress partial snapshots agent-tracing partial list

# Inspect a partial (use `inspect` directly — all flags work with partial IDs) agent-tracing inspect <partialOperationId> agent-tracing inspect <partialOperationId> -T agent-tracing inspect <partialOperationId> -p

# Clean up stale partial snapshots agent-tracing partial clean

# Map the context window composition of every LLM call (cm / map are aliases) agent-tracing ctx-map agent-tracing ctx-map <operationId|traceId|path.json> agent-tracing ctx-map --html # standalone report under .agent-tracing/_reports/ agent-tracing ctx-map --html out.html

# Re-issue a recorded LLM call against other models (see "replay" below) agent-tracing replay <operationId|traceId|path.json> agent-tracing replay <target> -s 4 -m openai/gpt-5,anthropic/claude-sonnet-5 agent-tracing replay <target> --all-steps ```

## replay — Re-issue a Frozen Call

A snapshot already freezes everything one LLM call saw: `steps[].contextEngine.output` is the exact message array sent to the model, `context.payload.tools` the toolset it could reach. `replay` sends that frozen payload back out with only the model swapped, so a difference in output is attributable to the model rather than to context assembly. If every model fails the same payload, the context is at fault; if some pass, it is model selection.

Available from both CLIs — `agent-tracing replay` (reads `LOBEHUB_JWT`) and `lh trace op replay` (uses the `lh login` session). Both need credentials because the call goes out through the LobeHub chat route.

```bash # One call: defaults to the last call_llm step and the model the op ran on lh trace op replay <operationId> lh trace op replay <operationId> -s 4 -m openai/gpt-5,anthropic/claude-sonnet-5 lh trace op replay <operationId> --judge "answers with a concrete file path"

# Every call of the operation lh trace op replay <operationId> --all-steps lh trace op replay <operationId> --all-steps --concurrency 8 ```

`--all-steps` answers the question the whole feature exists for: **take a run that succeeded, put another model on it, and see whether that model also gets the job done.** It ends in a PASS / FAIL verdict from an llm-rubric judge comparing the replayed outcome against the recorded one; pass `--judge "<criteria>"` to define success yourself instead of using the default rubric.

The judge scores the **outcome, not the route**. A model that solved the same problem by calling different tools has passed. The per-call tool comparison (`toolSignature`) is reported underneath as supporting evidence — where the run took another path — and never decides pass / fail. A final call that never reached the model is a FAIL, not a missing verdict.

Each call is replayed independently, against the payload the harness actually built for it, so a different answer at call 2 cannot contaminate call 4, a call that fails to reach the provider costs only itself, and the calls go out concurrently (4 at a time by default).

Chaining the nodes — feeding each replayed output into the next — was built and then removed. A trace cannot regenerate tool output, only hand back what was recorded, so the moment the model deviates there is no ground truth left; the run then measures nothing while still looking like it succeeded. Independent replay is the design, not a fallback.

Known limitation: **sampling parameters are not recorded.** The recorder stores the runtime step context under `context.payload`, not the provider request body, so temperature, `top_p`, `max_tokens`, penalties and reasoning config are absent from every trace written so far. Replays use the server's current defaults for those. Model-to-model comparisons stay valid (every target gets the same request); comparisons against the recorded output do not, for an operation that ran with non-default settings. `--temperature` / `--max-tokens` override explicitly. The CLI prints a `note` line whenever a replay is running without recorded parameters.

## ctx-map — Context Window Composition

`ctx-map` renders one row per `call_llm` step: the messages that call sent to the model, split into typed segments (system / injected block / user / reasoning / tool call / tool result) with width proportional to tokens, laid against the model's context window.

The second axis is what `ctx-lint` cannot show: each row is diffed against the previous call, so the longest identical **message prefix** — the part a provider's prefix cache can reuse — is marked, along with the first message that mutated and the tokens re-processed behind it. A small injected block carrying a relative timestamp (`1m ago` → `now`) invalidates every token after it, which shows up as a break marker early in the row.

Reading a row:

- **Block colors** encode role directly: orange system, green user, blue assistant, gray tool. Assistant reasoning, content, and tool calls use different steps of the same blue scale; framework-injected blocks use a lighter orange than the system prompt. Every value is a step index into a LobeHub scale vendored in `viewer/contextMapScales.ts`, with assignments in `viewer/contextMapPalette.ts`. The HTML report ships both themes and follows the system theme. - **Neutral message frames** group segments that belong to the same payload message. Every role uses the same frame color: the outline communicates structure only, while the block fill carries role and subtype semantics. The original framed layout uses padding inside each message and a small track gap between messages, keeping each payload message visually distinct. - **Fill** says whether the provider reused it: shaded `▓` (HTML: 60%-opaque hatch) was served from the prefix cache; solid `█` (HTML: flat) was re-processed by the model. - **The line under the track** is the cache ledger — a bracket / green band spanning exactly the cached prefix, then the break marker (`▲` in the terminal, a red rule through the track in HTML) at the column where reuse stopped, with the reason and the re-processed tokens.

| Flag | Short | Description | | --------------- | ----- | ---------------------------------------------------------- | | `--html [path]` | | Standalone HTML report (hover a segment for its content) | | `--width <n>` | `-w` | Track width in terminal columns | | `--window <n>` | | Override the model context window used as the track | | `--full-window` | | Always scale to the full window, never to the largest call | | `--json` | `-j` | Per-call segments + cache stats |

The track scales to the context window; when the window dwarfs the payloads (a 50k payload on a 1M window) it falls back to the largest call so the composition stays readable, and the header says which basis is in use. Analysis lives in `analysis/contextMap.ts` and is exported as `buildContextMap()` for downstream corpus work.

## Inspect Flag Reference

| Flag | Short | Description | Default Step | | ----------------- | ----- | ------------------------------------------------------------------------------------------------- | ------------ | | `--step <n>` | `-s` | Target a specific step | — | | `--messages` | `-m` | Messages context (CE input → params → LLM payload) | — | | `--tools` | `-t` | Tool calls & results (what agent invoked) | — | | `--events` | `-e` | Raw events (llm_start, llm_result, etc.) | — | | `--context` | `-c` | Runtime context & payload (raw) | — | | `--system-role` | `-r` | Full system role content | 0 | | `--env` | | Environment context | 0 | | `--payload` | `-p` | Context engine input overview (model, knowledge, tools summary, memory summary, platform context) | 0 | | `--payload-tools` | `-T` | Available tools detail (plugin manifests + LLM function definitions) | 0 | | `--memory` | `-M` | Full user memory (persona, identity, contexts, preferences, experiences) | 0 | | `--diff <n>` | `-d` | Diff against step N (use with `-r` or `--env`) | — | | `--msg <n>` | | Full content of message N from Final LLM Payload | — | | `--msg-input <n>` | | Full content of message N from Context Engine Input | — | | `--json` | `-j` | Output as JSON (combinable with any flag above) | — |

Flags marked "Default Step: 0" auto-select step 0 if `--step` is not provided. All flags support `latest` or omitted traceId.

## Typical Debug Workflow

```bash # 1. Trigger an agent operation in the dev UI

# 2. See the overview agent-tracing inspect

# 3. List all traces, get traceId agent-tracing list

# 4. Quick overview of what was fed into context engine agent-tracing inspect -p

# 5. Inspect a specific step's messages to see what was sent to the LLM agent-tracing inspect TRACE_ID -s 0 -m

# 6. Drill into a truncated message for full content agent-tracing inspect TRACE_ID -s 0 --msg 2

# 7. Check available tools vs actual tool calls agent-tracing inspect -T # available tools agent-tracing inspect -s 1 -t # actual tool calls & results

# 8. Inspect user memory injected into the conversation agent-tracing inspect -M

# 9. Diff system role between steps (multi-step agents) agent-tracing inspect TRACE_ID -r -d 2 ```

## Key Types

```typescript interface ExecutionSnapshot { traceId: string; operationId: string; model?: string; provider?: string; startedAt: number; completedAt?: number; completionReason?: 'done' | 'error' | 'interrupted' | 'max_steps' | 'cost_limit' | 'waiting_for_human'; totalSteps: number; totalTokens: number; totalCost: number; error?: { type: string; message: string }; steps: StepSnapshot[]; }

interface StepSnapshot { stepIndex: number; stepType: 'call_llm' | 'call_tool'; executionTimeMs: number; content?: string; // LLM output reasoning?: string; // Reasoning/thinking inputTokens?: number; outputTokens?: number; toolsCalling?: Array<{ apiName: string; identifier: string; arguments?: string }>; toolsResult?: Array<{ apiName: string; identifier: string; isSuccess?: boolean; output?: string; }>; messages?: any[]; // DB messages before step context?: { phase: string; payload?: unknown; stepContext?: unknown }; events?: Array<{ type: string; [key: string]: unknown }>; contextEngine?: { input?: unknown; // contextEngineInput minus messages + toolsConfig (reconstructible from baseline) output?: unknown; // processed messages array (final LLM payload) metadata?: unknown; // pipeline decision records (trim stats, cache gates, truncation counts) }; } ```

## --messages Output Structure

When using `--messages`, the output shows three sections (if context engine data is available):

1. **Context Engine Input** — DB messages passed to the engine, with `[0]`, `[1]`, ... indices. Use `--msg-input N` to view full content. 2. **Context Engine Params** — systemRole, model, provider, knowledge, tools, userMemory, etc. 3. **Final LLM Payload** — Processed messages after context engine (system date injection, user memory, history truncation, etc.), with `[0]`, `[1]`, ... indices. Use `--msg N` to view full content.

## Integration Points

- **Recording**: `apps/server/src/services/agentRuntime/AgentRuntimeService/executeStep/index.ts` — in `executeStep()`, after `saveAndPublishStep` builds `stepPresentationData`, writes partial snapshot in dev mode - **Context engine capture**: `apps/server/src/modules/AgentRuntime/RuntimeExecutors.ts` — in `call_llm` executor, after `serverMessagesEngine()` returns, calls `ctx.tracingContextEngine(input, output, metadata)`. `AgentRuntimeService.executeStep` buffers it per step and passes it to `traceRecorder.appendStep` as the typed `contextEngine` field (kept off the `events` array to stay out of Redis state). - **Store**: `FileSnapshotStore` reads/writes to `.agent-tracing/` relative to `process.cwd()`

Intended uses

  • Use Agent Tracing when this documented workflow matches the task.

Implementation notes

  • Review the source instructions and adapt inputs before running the workflow.

Try it, then tell us how it went

A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.