Skip to main content
Agent skill

Gke Ai Troubleshooting Tpu Metrics Monitoring

Monitors and troubleshoots GKE TPU workloads, nodes, and node pools using GKE system metrics and PromQL. Use when monitoring TensorCore duty cycle, TPU memory, node readiness, multi-host TPU node pool availability, host maintenance or preemption interruptions, and calculating MTTR or MTBI metrics for GKE TPUs. Don't use for general non-TPU GKE workload monitoring or non-metric TPU debugging.

Get started

Get the complete skill folder, including its scripts and reference files.

When to use this skill

Use Gke Ai Troubleshooting Tpu Metrics Monitoring when an AI agent needs a reusable SKILL.md workflow for this job: Monitors and troubleshoots GKE TPU workloads, nodes, and node pools using GKE system metrics and PromQL. Use when monitoring TensorCore duty cycle, TPU memory, node readiness, multi-host TPU node pool availability, host maintenance or preemption interruptions, and calculating MTTR or MTBI metrics for GKE TPUs. Don't use for general non-TPU GKE workload monitoring or non-metric TPU debugging.

When not to use it

Skip Gke Ai Troubleshooting Tpu Metrics Monitoring when the task is outside the coding category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.

How to install

Where will you use it?

Install scope
Claude Code folder setup
mkdir -p ~/.claude/skills/gke-ai-troubleshooting-tpu-metrics-monitoring
# Save SKILL.md to ~/.claude/skills/gke-ai-troubleshooting-tpu-metrics-monitoring/SKILL.md
  1. Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/gke-ai-troubleshooting-tpu-metrics-monitoring/SKILL.md. For a shared project, commit the folder at .claude/skills/gke-ai-troubleshooting-tpu-metrics-monitoring/ instead.
  2. Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
  3. For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.

Full install guide for Claude, Cursor, and Codex →

Try a starter prompt

After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.

Starter prompt
Use the Gke Ai Troubleshooting Tpu Metrics Monitoring skill for this task: Use Gke Ai Troubleshooting Tpu Metrics Monitoring when this documented workflow matches the task. Confirm any required inputs or files before making changes.

What supports this recommendation?

71/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.

Documentation
77/100
Practical detail
68/100
Source evidence
63/100
Source signals
72/100
Model compatibility evidence
  • claudeuntested

    No model-specific signal or recorded compatibility test was found.

  • chatgptuntested

    No model-specific signal or recorded compatibility test was found.

  • geminiuntested

    No model-specific signal or recorded compatibility test was found.

  • copilotuntested

    No model-specific signal or recorded compatibility test was found.

  • llamauntested

    No model-specific signal or recorded compatibility test was found.

  • perplexityuntested

    No model-specific signal or recorded compatibility test was found.

  • mistraluntested

    No model-specific signal or recorded compatibility test was found.

  • grokuntested

    No model-specific signal or recorded compatibility test was found.

Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.

What this skill does

# GKE TPU Metrics Monitoring Guide

This skill enables the agent to monitor GKE TPU workloads, nodes, and node pools using GKE system metrics. It helps diagnose if workload interruptions or performance issues are caused by underlying infrastructure.

## Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you MUST read [Usage Attribution](references/attribution.md). For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix `gcloud` with `CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-metrics-monitoring)"` and append `agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-metrics-monitoring)` to `User-Agent` headers.

## Step 0: Mandatory Context

Independently gather required context (such as cluster details or node pool names) using available GKE and Cloud tools, or use the provided `{variable}` placeholders:

- `{project_id}`: The GCP Project ID. - `{cluster_name}`: The GKE Cluster Name. - `{location}`: The GKE Cluster Location (region or zone). - `{node_name}`: (Optional) The name of the specific GKE node. - `{node_pool_name}`: (Optional) The name of the GKE node pool.

---

## Diagnostic Steps

### Step 1: Verify TPU Runtime Metrics Configuration [Low Risk] [Auto]

Before analyzing runtime metrics, verify that the workload is configured to export them. This ensures the cluster and container environment are set up for automated metric scraping and visibility into accelerator health.

- **Action**: Verify that the Pod specification and cluster meet the following prerequisites: - `containerPort: 8431` exposed on the TPU container (required for Prometheus metric scraping). - JAX version `0.4.14` or later if using JAX (earlier versions do not export runtime metrics). - GKE version is `1.27.4-gke.900` or later (required for TPU runtime metric support). - GKE System Metrics are enabled on the cluster (required for Cloud Monitoring ingestion).

### Step 2: Monitor TPU Runtime Metrics [Low Risk] [Auto]

If configured correctly, the following metrics are available in Cloud Monitoring (monitored resources `k8s_node` and `k8s_container`):

- **Container Metrics**: - `kubernetes.io/container/accelerator/duty_cycle`: Percentage of time over the past sampling period (60 seconds) during which the TensorCores were actively processing on a TPU chip. - `kubernetes.io/container/accelerator/memory_used`: Amount of accelerator memory allocated in bytes. - `kubernetes.io/container/accelerator/memory_total`: Total accelerator memory in bytes. - **Node Metrics**: - `kubernetes.io/node/accelerator/duty_cycle` - `kubernetes.io/node/accelerator/memory_used` - `kubernetes.io/node/accelerator/memory_total`

### Step 3: Check Node Status Condition [Low Risk] [Auto]

Query the status condition of GKE nodes (GKE version `1.32.1-gke.1357001` or later).

- **PromQL Query (Check if a specific node is Ready)**: ```promql kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", node_name="{node_name}", condition="Ready", status="True"} ``` - **PromQL Query (List nodes with non-Ready conditions that are True)**: ```promql kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", condition!="Ready", status="True"} ``` - **PromQL Query (List nodes that are NOT Ready)**: ```promql kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", condition="Ready", status="False"} ``` - **PromQL Query (Fleet-wide node status)**: ```promql avg by (condition,status)(avg_over_time(kubernetes_io:node_status_condition{monitored_resource="k8s_node"}[5m])) ```

### Step 4: Check Node Pool Status [Low Risk] [Auto]

Query the status of multi-host TPU node pools.

- **PromQL Query (Verify if a specific node pool is Running)**: ```promql kubernetes_io:node_pool_status{monitored_resource="k8s_node_pool", cluster_name="{cluster_name}", node_pool_name="{node_pool_name}", status="Running"} ``` - **PromQL Query (Monitor node pools grouped by status)**: ```promql count by (status)(count_over_time(kubernetes_io:node_pool_status{monitored_resource="k8s_node_pool"}[5m])) ``` _Possible statuses_: `Provisioning`, `Running`, `Error`, `Reconciling`, `Stopping`.

### Step 5: Check Node Pool Availability [Low Risk] [Auto]

Query if all nodes in a multi-host TPU node pool are available.

- **PromQL Query (Check availability over time)**: ```promql avg by (node_pool_name)(avg_over_time(kubernetes_io:node_pool_multi_host_available{monitored_resource="k8s_node_pool", cluster_name="{cluster_name}"}[5m])) ``` _Value_: `1` (True, all nodes available) or `0` (False, some nodes unavailable).

### Step 6: Analyze Node Interruptions [Low Risk] [Auto]

Query the count of interruptions for GKE nodes.

- **PromQL Query (Breakdown of interruptions and causes)**: ```promql sum by (interruption_type,interruption_reason)(sum_over_time(kubernetes_io:node_interruption_count{monitored_resource="k8s_node"}[5m])) ``` _Interruption Types_: `TerminationEvent`, `MaintenanceEvent`, `PreemptionEvent`. _Interruption Reasons_: `HostError`, `Eviction`, `AutoRepair`. - **PromQL Query (Filter for Host Maintenance events)**: ```promql sum by (interruption_type,interruption_reason)(sum_over_time(kubernetes_io:node_interruption_count{monitored_resource="k8s_node", interruption_reason="HW/SW Maintenance"}[5m])) ``` - **PromQL Query (Interruption count aggregated by node pool)**: ```promql sum by (node_pool_name,interruption_type,interruption_reason)(sum_over_time(kubernetes_io:node_pool_interruption_count{monitored_resource="k8s_node_pool", interruption_reason="HW/SW Maintenance", node_pool_name="{node_pool_name}"}[5m])) ```

### Step 7: Calculate Recovery and Interruption Metrics [Low Risk] [Auto]

Calculate Mean Time to Recovery (MTTR) and Mean Time Between Interruptions (MTBI) over the last 7 days.

- **PromQL Query (MTTR - Mean Time to Recovery)**: ```promql sum(sum_over_time(kubernetes_io:node_pool_accelerator_times_to_recover_sum{monitored_resource="k8s_node_pool", cluster_name="{cluster_name}"}[7d])) / sum(sum_over_time(kubernetes_io:node_pool_accelerator_times_to_recover_count{monitored_resource="k8s_node_pool",cluster_name="{cluster_name}"}[7d])) ``` - **PromQL Query (MTBI - Mean Time Between Interruptions)**: ```promql sum(count_over_time(kubernetes_io:node_memory_total_bytes{monitored_resource="k8s_node", node_name=~"gke-tpu.*|gk3-tpu.*", cluster_name="{cluster_name}"}[7d])) / sum(sum_over_time(kubernetes_io:node_interruption_count{monitored_resource="k8s_node", node_name=~"gke-tpu.*|gk3-tpu.*", cluster_name="{cluster_name}"}[7d])) ```

### Step 8: Monitor TPU Host Metrics [Low Risk] [Auto]

For GKE version `1.28.1-gke.1066000` or later, monitor TPU host performance.

- **Container Metrics**: - `kubernetes.io/container/accelerator/tensorcore_utilization`: Current percentage of the TensorCore that is utilized. - `kubernetes.io/container/accelerator/memory_bandwidth_utilization`: Current percentage of the accelerator memory bandwidth that is being used. - **Node Metrics**: - `kubernetes.io/node/accelerator/tensorcore_utilization` - `kubernetes.io/node/accelerator/memory_bandwidth_utilization`

Intended uses

  • Use Gke Ai Troubleshooting Tpu Metrics Monitoring when this documented workflow matches the task.

Implementation notes

  • Review the source instructions and adapt inputs before running the workflow.

Try it, then tell us how it went

A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.