Gke Ai Troubleshooting Handle Disruption Gpu Tpu
Diagnoses, predicts, and mitigates node disruptions during Compute Engine host maintenance and hardware or software maintenance events for GPU and TPU workloads on GKE. Use when diagnosing node disruptions, predicting host maintenance events on GPU/TPU nodepools, inspecting node interruption PromQL metrics, auditing node taints, or configuring workload protection strategies (graceful termination, opportunistic maintenance, PodDisruptionBudgets). Don't use for general GKE cluster creation, network policy configuration, or non-disruption workload deployment.
Get the complete skill folder, including its scripts and reference files.
When to use this skill
Use Gke Ai Troubleshooting Handle Disruption Gpu Tpu when an AI agent needs a reusable SKILL.md workflow for this job: Diagnoses, predicts, and mitigates node disruptions during Compute Engine host maintenance and hardware or software maintenance events for GPU and TPU workloads on GKE. Use when diagnosing node disruptions, predicting host maintenance events on GPU/TPU nodepools, inspecting node interruption PromQL metrics, auditing node taints, or configuring workload protection strategies (graceful termination, opportunistic maintenance, PodDisruptionBudgets). Don't use for general GKE cluster creation, network policy configuration, or non-disruption workload deployment.
When not to use it
Skip Gke Ai Troubleshooting Handle Disruption Gpu Tpu when the task is outside the coding category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.
How to install
Where will you use it?
mkdir -p ~/.claude/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu # Save SKILL.md to ~/.claude/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu/SKILL.md
- Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu/SKILL.md. For a shared project, commit the folder at .claude/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu/ instead.
- Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
- For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.
Full install guide for Claude, Cursor, and Codex →
Try a starter prompt
After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.
Use the Gke Ai Troubleshooting Handle Disruption Gpu Tpu skill for this task: Use Gke Ai Troubleshooting Handle Disruption Gpu Tpu when this documented workflow matches the task. Confirm any required inputs or files before making changes.
What supports this recommendation?
71/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.
- Documentation
- 77/100
- Practical detail
- 68/100
- Source evidence
- 63/100
- Source signals
- 72/100
Model compatibility evidence
- claudeuntested
No model-specific signal or recorded compatibility test was found.
- chatgptuntested
No model-specific signal or recorded compatibility test was found.
- geminiuntested
No model-specific signal or recorded compatibility test was found.
- copilotuntested
No model-specific signal or recorded compatibility test was found.
- llamauntested
No model-specific signal or recorded compatibility test was found.
- perplexityuntested
No model-specific signal or recorded compatibility test was found.
- mistraluntested
No model-specific signal or recorded compatibility test was found.
- grokuntested
No model-specific signal or recorded compatibility test was found.
Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.
What this skill does
# Handle Disruption on GPUs and TPUs Troubleshooting
## Attribution
**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you MUST read [Usage Attribution](references/attribution.md). For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix `gcloud` with `CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-handle-disruption-gpu-tpu)"` and append `agent-skills/1.0 (skill:gke-ai-troubleshooting-handle-disruption-gpu-tpu)` to `User-Agent` headers.
## 🔍 Diagnostic Workflow
### Step 0: Context Acquisition
- **Mandatory**: When a user asks to debug or investigate an actual workload disruption, node crash, or unexpected restart without providing complete cluster details, you MUST immediately halt and request all missing mandatory parameters (`project_id`, `location`, `cluster_name`, `timestamp`) BEFORE delivering theories or general diagnostic commands. Only skip context acquisition if the user explicitly requests a generic reusable runbook or provides a complete static telemetry/log dump for offline analysis. - **Optional**: `node_name`, `workload_name`, `workload_namespace`, `nodepool_name`.
### Step 1: [Low Risk] Check for Upcoming Scheduled Maintenance
- **Action**: Propose running `kubectl` to check if nodes have the scheduled maintenance label indicating an upcoming disruption. - **Example Command**:
```bash kubectl get nodes -l cloud.google.com/scheduled-maintenance-time -L cloud.google.com/scheduled-maintenance-time ```
- **Interpretation**: The `SCHEDULED-MAINTENANCE-TIME` column shows the Unix epoch time when the VM is scheduled for maintenance. If this label exists, a disruption is guaranteed to occur.
### Step 2: [Low Risk] Investigation via Cloud Monitoring (PromQL)
- **Action**: Call any available monitoring tool or provide PromQL for manual verification. - **Mandatory Monitoring Rule**: Whenever recommending follow-up monitoring or interruption tracking over time, you MUST explicitly present a **PromQL** query using the metric `kubernetes_io:node_interruption_count` filtered by `interruption_reason="HW/SW Maintenance"`. Do not suggest general Cloud Monitoring dashboards or Metrics Explorer without providing this specific PromQL metric expression. - **Example Query**:
```promql # Fetch host maintenance events for nodes sum by (interruption_type,interruption_reason)( sum_over_time( kubernetes_io:node_interruption_count{monitored_resource="k8s_node", interruption_reason="HW/SW Maintenance"}[${__interval}])) ```
```promql # See the interruption count aggregated by node pool sum by (node_pool_name,interruption_type,interruption_reason)( sum_over_time( kubernetes_io:node_pool_interruption_count{monitored_resource="k8s_node_pool", interruption_reason="HW/SW Maintenance", node_pool_name="{nodepool_name}" }[${__interval}])) ```
- **Interpretation**: If `kubernetes_io:node_interruption_count` shows values > 0 for `interruption_reason="HW/SW Maintenance"`, it indicates the underlying Compute Engine VM was interrupted due to scheduled host maintenance.
### Step 3: [Low Risk] Investigation via Cloud Logging & Node Taints
- **Action**: Call `query_logs` or instruct the user to filter their GKE logs for active host maintenance events, and check node taints. - **Guidance**: Look for occurrences in Cloud Logging where `cloud.google.com/active-node-maintenance` is set to `ONGOING`. To check if GKE has cordoned the terminating node to prevent new workloads from being scheduled, verify whether the `cloud.google.com/impending-node-termination:NoSchedule` taint is present (either in GKE event logs or directly via `kubectl describe node`). - **Interpretation**: - `cloud.google.com/active-node-maintenance` set to `ONGOING` means workloads are actively being stopped by GKE due to host maintenance. - `cloud.google.com/impending-node-termination:NoSchedule` taint means GKE has cordoned the node to prevent new Pods from being scheduled on the terminating node. DO NOT recommend tolerating this taint.
### Step 4: Conclusion and Resolution
- **Action**: Provide a summary of findings to the user and suggest appropriate mitigation strategies if host maintenance events were confirmed or scheduled. - **Reporting Rule**: Signal Only. Report high-signal information indicating that the disruption was caused by Compute Engine host maintenance, specifically affecting the underlying GPU/TPU nodes. DO NOT dump raw logs. - **Negative Findings Rule-Out**: If node scheduled-maintenance labels, PromQL interruption counts, and active maintenance logs all return negative/empty results, definitively conclude that Compute Engine host maintenance did NOT cause the disruption. Direct the user to investigate application-level causes (such as OOMKill events, CUDA runtime errors, or resource limits) and do not propose host maintenance mitigations as the primary resolution. - **Mandatory Workload Protection Triad**: Whenever host maintenance is identified or anticipated on GPU/TPU nodes, consistently recommend all three complementary mitigations together: 1. **Configure Graceful Termination**: For workloads that need time to save state (e.g., ML frameworks checkpointing via Orbax), follow the guide to [Enable disruption handling](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/handle-disruption-gpu-tpu#enabling-handling) and set `spec.terminationGracePeriodSeconds` (up to 60 minutes) to handle the `SIGTERM` signal before node shutdown. 2. **Enable Opportunistic Maintenance**: To automatically trigger maintenance when GKE detects that GPU/TPU nodes are idle, configure [Opportunistic Maintenance](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/handle-disruption-gpu-tpu#opportunistic-maintenance). 3. **Configure PodDisruptionBudgets (PDBs)**: Ensure your workload uses a `PodDisruptionBudget` to maintain `minAvailable` replicas during evictions and disruptions.
Intended uses
- Use Gke Ai Troubleshooting Handle Disruption Gpu Tpu when this documented workflow matches the task.
Implementation notes
- Review the source instructions and adapt inputs before running the workflow.
Try it, then tell us how it went
A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.