Skip to main content
Agent skill

Gke Ai Troubleshooting Tpu Vbar Oom

Diagnoses and prevents vbarcontrolagent segfaults, out-of-memory (OOM) errors, and TPU device initialization failures on TPU v6e nodes in GKE caused by race conditions during TPU device resets or high-frequency metrics polling. Use when troubleshooting vbarcontrolagent crashes, memory cgroup OOMs in serial console logs, tpu-device-plugin metrics checksum corruption errors, or custom TPU metrics collection conflicts on GKE TPU v6e nodes. Don't use for general non-TPU container OOM troubleshooting or standard GKE node lifecycle operations.

Get started

Get the complete skill folder, including its scripts and reference files.

When to use this skill

Use Gke Ai Troubleshooting Tpu Vbar Oom when an AI agent needs a reusable SKILL.md workflow for this job: Diagnoses and prevents vbarcontrolagent segfaults, out-of-memory (OOM) errors, and TPU device initialization failures on TPU v6e nodes in GKE caused by race conditions during TPU device resets or high-frequency metrics polling. Use when troubleshooting vbarcontrolagent crashes, memory cgroup OOMs in serial console logs, tpu-device-plugin metrics checksum corruption errors, or custom TPU metrics collection conflicts on GKE TPU v6e nodes. Don't use for general non-TPU container OOM troubleshooting or standard GKE node lifecycle operations.

When not to use it

Skip Gke Ai Troubleshooting Tpu Vbar Oom when the task is outside the creativity category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.

How to install

Where will you use it?

Install scope
Claude Code folder setup
mkdir -p ~/.claude/skills/gke-ai-troubleshooting-tpu-vbar-oom
# Save SKILL.md to ~/.claude/skills/gke-ai-troubleshooting-tpu-vbar-oom/SKILL.md
  1. Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/gke-ai-troubleshooting-tpu-vbar-oom/SKILL.md. For a shared project, commit the folder at .claude/skills/gke-ai-troubleshooting-tpu-vbar-oom/ instead.
  2. Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
  3. For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.

Full install guide for Claude, Cursor, and Codex →

Try a starter prompt

After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.

Starter prompt
Use the Gke Ai Troubleshooting Tpu Vbar Oom skill for this task: Use Gke Ai Troubleshooting Tpu Vbar Oom when this documented workflow matches the task. Confirm any required inputs or files before making changes.

What supports this recommendation?

71/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.

Documentation
77/100
Practical detail
68/100
Source evidence
63/100
Source signals
72/100
Model compatibility evidence
  • claudeuntested

    No model-specific signal or recorded compatibility test was found.

  • chatgptuntested

    No model-specific signal or recorded compatibility test was found.

  • geminiuntested

    No model-specific signal or recorded compatibility test was found.

  • copilotuntested

    No model-specific signal or recorded compatibility test was found.

  • llamauntested

    No model-specific signal or recorded compatibility test was found.

  • perplexityuntested

    No model-specific signal or recorded compatibility test was found.

  • mistraluntested

    No model-specific signal or recorded compatibility test was found.

  • grokuntested

    No model-specific signal or recorded compatibility test was found.

Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.

What this skill does

# TPU Connection Failure and VBAR OOM Troubleshooting

Use this skill to systematically diagnose and prevent `vbar_control_agent` segfaults and Out-Of-Memory (OOM) errors on TPU v6e nodes.

## Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you MUST read [Usage Attribution](references/attribution.md). For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix `gcloud` with `CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-vbar-oom)"` and append `agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-vbar-oom)` to `User-Agent` headers.

## ⚠️ Prerequisites

- Cloud Logging must be enabled for the project. - Access to the project and cluster via `gcloud` or equivalent tool.

## 🔍 Diagnostic Workflow

### Step 0: Context Acquisition & Time Window Definition

Independently gather required context using available GCP/GKE tools or use the provided `{variable}` placeholders:

- `{project_id}`: The GCP Project ID (e.g., `customer-ai-project-123`). - `{cluster_name}`: The GKE Cluster Name (e.g., `tpu-cluster-prod`). - `{node_name}`: The Node Name or Instance ID (e.g., `tpu-node-1`). - `{workload_name}`: The Workload Name / JobSet Name (e.g., `my-training-job-456`). - `{namespace}`: The Workload Namespace. - `{issue_time}`: The timestamp of the issue (e.g., `2026-04-14T20:00:00Z`).

#### Time Handling & Execution Rules

1. **Window Calculation**: If an issue timestamp `{issue_time}` is provided, calculate the query time window as `[{issue_time} - 30m]` to `[{issue_time} + 30m]`. - Let `{start_time}` = `{issue_time} - 30m` - Let `{end_time}` = `{issue_time} + 30m` 2. **Informational vs. Live Execution**: If the user request is informational or query-formulation (e.g. "How can I check...", "How do I determine..."), or if live GCP project resources are not actively targetable, directly output the calculated time window, log names, and Cloud Logging filter templates without attempting live log execution commands.

### Step 1: Check for `vbar_control_agent` OOMs

Look for specific `out of memory` messages from `vbar_control_agent` in serial console logs (`serialconsole.googleapis.com%2fserial_port_1_output`).

- **Tool to use**: `query_logs` (for live diagnostics) - **Filter Templates**:

**Serial Console Logs (OOMs):**

```sql logName="projects/{project_id}/logs/serialconsole.googleapis.com%2fserial_port_1_output" AND labels."compute.googleapis.com/resource_name"="{node_name}" AND SEARCH(text_payload, "Memory cgroup out of memory: Killed process .* (vbar_control_ag)") AND timestamp >= "{start_time}" AND timestamp <= "{end_time}" ```

- **Logic**: Presence of `Memory cgroup out of memory` messages related to `vbar_control_agent`. Stack traces pointing to `libtpu::tpunetd::VBARControlHelper::MetricsReadFromVBAR` are a strong indicator. - **Automation**: Proceed to next step automatically after reporting findings. - **Reference**: See `references/failure_signatures.md` for example log patterns.

### Step 2: Investigate `tpu-device-plugin` Metrics Fetch Failures [Low Risk]

Check if `tpu-device-plugin` is reporting metric fetch failures.

- **Tool to use**: `query_logs` - **Filter Template**:

```sql resource.type="k8s_container" AND resource.labels.project_id="{project_id}" AND resource.labels.cluster_name="{cluster_name}" AND resource.labels.container_name="tpu-device-plugin" AND severity=ERROR AND textPayload:"metrics fetch failed for .* deviceID and .* device path with error: checksum didn't match with the metrics data. Corrupt data found" AND timestamp >= "{start_time}" AND timestamp <= "{end_time}" ```

- **Logic**: Errors indicating "metrics fetch failed" with "checksum didn't match" suggest vBAR memory corruption. - **Automation**: Proceed to next step automatically after reporting findings.

### Step 3: Check for Custom Metrics Collection Usage [Low Risk]

Inspect cluster configurations, workloads, or container specs to determine if custom TPU metrics collection mechanisms are deployed.

- **Action**: Check if custom scripts or agents (e.g., using `libtpu.sdk.tpumonitoring`) are deployed that frequently query `GetHostMetrics` from `vBAR Control Agent`. - **Verification Commands**:

- **Kubectl Search (Inspect workload env/specs)**:

```bash kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}' ```

- **Log Search Filter (`query_logs`)**:

```sql resource.type="k8s_container" AND resource.labels.project_id="{project_id}" AND resource.labels.cluster_name="{cluster_name}" AND textPayload:"libtpu.sdk.tpumonitoring" AND timestamp >= "{start_time}" AND timestamp <= "{end_time}" ```

- **Logic**: Confirmation of custom metrics collection helps confirm the race condition hypothesis.

## 🛠️ Resolution Workflow

### Resolution 1: Temporarily Disable Custom Metrics Collection [High Risk]

If a custom metrics collection agent is identified, recommend disabling it.

- **Action**: Recommend disabling the custom metrics collector. - **Justification**: Prevents reads from vBAR during device resets, stopping crashes and OOMs.

### Resolution 2: Await `vbar_control_agent` Resiliency Update [Low Risk]

Advise that a permanent fix will be available in a future GKE version.

- **Action**: Recommend upgrading GKE when the fix is available. - **Justification**: The updated agent will be resilient to memory corruption and gracefully handle reads from unbound vBARs.

## 📋 copypaste checklist

- [ ] Acquire context and compute `[{start_time}, {end_time}]` window. - [ ] Check for `vbar_control_agent` segfaults and OOMs using `query_logs`. - [ ] Investigate `tpu-device-plugin` failures using `query_logs`. - [ ] Inspect for custom metrics collection usage. - [ ] Advise disabling custom metrics collection if applicable. - [ ] Advise awaiting resiliency update.

Intended uses

  • Use Gke Ai Troubleshooting Tpu Vbar Oom when this documented workflow matches the task.

Implementation notes

  • Review the source instructions and adapt inputs before running the workflow.

Try it, then tell us how it went

A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.