Skip to main content
Agent skill

Cloud Trace Querying

Query Cloud Trace spans, filter by latency thresholds or error status, correlate distributed traces with Cloud Logging, and diagnose latency bottlenecks across Google Cloud services. Use when investigating slow requests, analyzing trace hierarchies, or resolving latency regressions. Do NOT use for querying non-GCP telemetry or database query optimization outside Cloud Trace.

Get started

Get the complete skill folder, including its scripts and reference files.

When to use this skill

Use Cloud Trace Querying when an AI agent needs a reusable SKILL.md workflow for this job: Query Cloud Trace spans, filter by latency thresholds or error status, correlate distributed traces with Cloud Logging, and diagnose latency bottlenecks across Google Cloud services. Use when investigating slow requests, analyzing trace hierarchies, or resolving latency regressions. Do NOT use for querying non-GCP telemetry or database query optimization outside Cloud Trace.

When not to use it

Skip Cloud Trace Querying when the task is outside the coding category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.

How to install

Where will you use it?

Install scope
Claude Code folder setup
mkdir -p ~/.claude/skills/cloud-trace-querying
# Save SKILL.md to ~/.claude/skills/cloud-trace-querying/SKILL.md
  1. Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/cloud-trace-querying/SKILL.md. For a shared project, commit the folder at .claude/skills/cloud-trace-querying/ instead.
  2. Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
  3. For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.

Full install guide for Claude, Cursor, and Codex →

Try a starter prompt

After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.

Starter prompt
Use the Cloud Trace Querying skill for this task: Use Cloud Trace Querying when this documented workflow matches the task. Confirm any required inputs or files before making changes.

What supports this recommendation?

71/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.

Documentation
77/100
Practical detail
68/100
Source evidence
63/100
Source signals
72/100
Model compatibility evidence
  • claudeuntested

    No model-specific signal or recorded compatibility test was found.

  • chatgptuntested

    No model-specific signal or recorded compatibility test was found.

  • geminiuntested

    No model-specific signal or recorded compatibility test was found.

  • copilotuntested

    No model-specific signal or recorded compatibility test was found.

  • llamauntested

    No model-specific signal or recorded compatibility test was found.

  • perplexityuntested

    No model-specific signal or recorded compatibility test was found.

  • mistraluntested

    No model-specific signal or recorded compatibility test was found.

  • grokuntested

    No model-specific signal or recorded compatibility test was found.

Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.

What this skill does

# Cloud Trace Querying Skill (`cloud-trace-querying`)

This skill equips the agent to interact with Cloud Trace by providing standard query patterns, diagnostic workflows for latency and errors, and tools to fetch and inspect trace hierarchies.

> [!IMPORTANT] > **AUTONOMOUS TOOL EXECUTION MANDATE**: > You MUST execute the tool scripts directly in bash to query telemetry. > Do NOT just describe query strings or write Python scripts. Run the provided bash tools directly and report the exact results.

## Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you MUST read [Usage Attribution](references/attribution.md). For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix `gcloud` with `CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:cloud-trace-querying)"` and append `agent-skills/1.0 (skill:cloud-trace-querying)` to `User-Agent` headers.

## Prerequisites & Setup 1. **Google Cloud SDK**: Verify `gcloud` is installed ([Installation Guide](https://docs.cloud.google.com/sdk/docs/install-sdk)). 2. **Authentication**: Authenticate CLI and application default credentials: ```bash gcloud auth login && gcloud auth application-default login ``` 3. **Project & Billing**: Identify the target `<PROJECT_ID>` (from the user's prompt or via `bash scripts/get_trace_project/run.sh`), ensure an active billing account is attached, and configure the project if not already set: ```bash gcloud config set project <PROJECT_ID> ``` 4. **API Enablement**: Verify Cloud Trace and Cloud Logging APIs are enabled: ```bash gcloud services enable cloudtrace.googleapis.com logging.googleapis.com ```

## Quickstart Workflows

### 1. Identify Google Cloud Project ID If the user specified a Google Cloud project ID in their prompt, use it.

If they did not, ALWAYS determine the Google Cloud project ID by running:

```bash bash scripts/get_trace_project/run.sh ```

### 2. Search Traces (Latency & Errors) To find traces matching specific criteria, such as a root span of `/v1/booking/book` with a duration of at least 1.5s:

```bash bash scripts/search_traces/run.sh --filter 'root:/v1/booking/book latency:1.5s' --projects '<PROJECT_ID>' ```

*Note: `--filter` is required. To list traces without specific filtering, pass `--filter ''` or `--filter 'root:'`.* *Filter syntax: The Cloud Trace API uses `latency:<duration>` (such as `latency:1.5s` for duration >= 1.5s), which corresponds to "Span duration" in Trace Explorer.* *The script outputs JSON where each cluster's `examples` array contains sample trace objects with `trace_id`, `duration_seconds`, span count, and error status.*

### 3. Fetch Entire Trace Hierarchy To view all spans and status codes/labels within a trace:

```bash bash scripts/fetch_entire_trace/run.sh --trace-id '<TRACE_ID>' --project '<PROJECT_ID>' ```

### 4. Fetch Correlated Application Logs To retrieve application log entries correlated with a trace ID:

```bash bash scripts/fetch_related_logs/run.sh --trace-id '<TRACE_ID>' --trace-projects '<PROJECT_ID>' ```

*Application log entries are useful when inspecting stack traces, exception messages, and log events emitted during span execution. When diagnosing errors or service crashes, inspect the returned log entries for `severity: "ERROR"` to confirm root causes or uncover unhandled exceptions.* *Note: `fetch_related_logs` queries the default log scope (`_Default`), searching across configured log views in the target project. An empty response can occur if: (1) no log entries match the criteria, (2) the caller lacks IAM permissions to view the matching entries, or (3) matching entries reside in log views outside the searched log scope.*

### 5. Fetch Specific Trace Span To inspect attributes and metadata of a single span within a trace:

```bash bash scripts/fetch_trace_span/run.sh --trace-id '<TRACE_ID>' --span-id '<SPAN_ID>' --project '<PROJECT_ID>' ```

### 6. Generate Cloud Console Deep Links To generate direct Cloud Console URLs for browser inspection of traces or correlated logs:

```bash bash scripts/generate_links/run.sh --trace-id '<TRACE_ID>' --project '<PROJECT_ID>' [--span-id '<SPAN_ID>'] ```

### 7. Format Output - **Terminal / CLI**: Render span trees as indented ASCII timelines using `|--`. - **UI Presentation**: Render as A2UI JSON graph structures when requested for UI.

### 8. Upstream Service Error Handling If the Cloud Trace API or Cloud Logging API returns an upstream HTTP service error (such as `500 Internal Server Error`, `503 Service Unavailable`, or `GoogleAPICallError`), report the upstream service outage directly to the user. Do not attempt to inspect virtual environments, modify internal scripts, or debug local libraries.

## Reference Documentation - **Master Index**: [Reference Documentation Index](./references/index.md)

### Operational Workflows - [Troubleshoot Latency Workflow](./references/troubleshoot-latency.md) - [Correlated Logs Workflow](./references/correlated-logs.md) - [Trace Querying Decision Tree](./references/decision-tree.md) - [Console Deep Links](./references/console-deep-links.md) - [Workflows Overview](./references/workflows-overview.md)

### Query Syntax & Tools - [Filter Syntax](./references/filter-syntax.md) (`root:`, `latency:`, `+span:`, `error:true`) - [API Limits and Constraints](./references/api-limits.md) - [Observability Analytics (BigQuery SQL)](./references/querying-observability-analytics.md) - [Cloud Trace MCP Tools](./references/querying-mcp.md)

### Semantic Conventions - [Conventions Overview](./references/conventions.md) - [OpenTelemetry Core](./references/conventions-otel.md) - [OpenTelemetry HTTP](./references/conventions-otel-http.md) - [OpenTelemetry gRPC & RPC](./references/conventions-otel-grpc.md) - [OpenTelemetry GenAI](./references/conventions-otel-genai.md) - [GCP App Hub & Service Conventions](./references/conventions-gcp.md) - [GCP Legacy Labels](./references/conventions-gcp-legacy.md)

### Concepts & Architecture - [Observability Scopes](./references/observability-scopes.md) - [Distributed Traces Concept](./references/concepts-trace.md) - [Spans Concept](./references/concepts-span.md) - [Logs-to-Trace Correlation](./references/logs-trace-correlation.md) - [Knowledge Overview](./references/knowledge-overview.md)

### Templates & Assets - [ASCII Formatting Templates](./assets/formatting/ascii) - [A2UI Formatting Templates](./assets/formatting/a2ui) - [Query Scenarios](./assets/queries)

## Official Documentation Links - [Cloud Trace Documentation](https://docs.cloud.google.com/trace/docs) - [Finding and Viewing Traces](https://docs.cloud.google.com/trace/docs/finding-traces) - [OpenTelemetry Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/)

Intended uses

  • Use Cloud Trace Querying when this documented workflow matches the task.

Implementation notes

  • Review the source instructions and adapt inputs before running the workflow.

Try it, then tell us how it went

A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.