Skip to main content
Agent skill

Gke Workload Scaling Troubleshooting

Diagnoses GKE HorizontalPodAutoscaler (HPA) failures — metrics showing as <unknown>, FailedGetResourceMetric / FailedGetScale / FailedComputeMetricsReplicas events, missing Pod resource requests, custom/external metrics-pipeline breakage (FailedGetExternalMetric / FailedGetCustomMetric, unavailable metrics adapter, control-plane firewall blocking the adapter), HPA that won't scale up or down (tolerance / stabilization window / unavailable rate metrics), scale-to/from-zero problems, and slow HPA reaction on large clusters. Use when an HPA isn't scaling a workload as expected or reports metric errors. Don't use for configuring or authoring new HPA/VPA objects or scaling best practices (see the gke-workload-scaling skill), or for Cluster Autoscaler / node-pool sizing.

Get started

Get the complete skill folder, including its scripts and reference files.

When to use this skill

Use Gke Workload Scaling Troubleshooting when an AI agent needs a reusable SKILL.md workflow for this job: Diagnoses GKE HorizontalPodAutoscaler (HPA) failures — metrics showing as <unknown>, FailedGetResourceMetric / FailedGetScale / FailedComputeMetricsReplicas events, missing Pod resource requests, custom/external metrics-pipeline breakage (FailedGetExternalMetric / FailedGetCustomMetric, unavailable metrics adapter, control-plane firewall blocking the adapter), HPA that won't scale up or down (tolerance / stabilization window / unavailable rate metrics), scale-to/from-zero problems, and slow HPA reaction on large clusters. Use when an HPA isn't scaling a workload as expected or reports metric errors. Don't use for configuring or authoring new HPA/VPA objects or scaling best practices (see the gke-workload-scaling skill), or for Cluster Autoscaler / node-pool sizing.

When not to use it

Skip Gke Workload Scaling Troubleshooting when the task is outside the coding category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.

How to install

Where will you use it?

Install scope
Claude Code folder setup
mkdir -p ~/.claude/skills/gke-workload-scaling-troubleshooting
# Save SKILL.md to ~/.claude/skills/gke-workload-scaling-troubleshooting/SKILL.md
  1. Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/gke-workload-scaling-troubleshooting/SKILL.md. For a shared project, commit the folder at .claude/skills/gke-workload-scaling-troubleshooting/ instead.
  2. Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
  3. For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.

Full install guide for Claude, Cursor, and Codex →

Try a starter prompt

After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.

Starter prompt
Use the Gke Workload Scaling Troubleshooting skill for this task: Use Gke Workload Scaling Troubleshooting when this documented workflow matches the task. Confirm any required inputs or files before making changes.

What supports this recommendation?

71/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.

Documentation
77/100
Practical detail
68/100
Source evidence
63/100
Source signals
72/100
Model compatibility evidence
  • claudeuntested

    No model-specific signal or recorded compatibility test was found.

  • chatgptuntested

    No model-specific signal or recorded compatibility test was found.

  • geminiuntested

    No model-specific signal or recorded compatibility test was found.

  • copilotuntested

    No model-specific signal or recorded compatibility test was found.

  • llamauntested

    No model-specific signal or recorded compatibility test was found.

  • perplexityuntested

    No model-specific signal or recorded compatibility test was found.

  • mistraluntested

    No model-specific signal or recorded compatibility test was found.

  • grokuntested

    No model-specific signal or recorded compatibility test was found.

Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.

What this skill does

# GKE Workload Scaling Troubleshooting Skill

Use this skill to systematically diagnose and resolve **HorizontalPodAutoscaler (HPA) failures** on GKE — metrics reported as `<unknown>`, `FailedGet*` events, missing resource requests, custom/external metrics-pipeline breakage, HPA that refuses to scale up or down, scale-to/from-zero issues, and slow HPA reaction on large clusters. This skill operates non-interactively and enforces a read-only diagnostics boundary before proposing manifest or configuration corrections.

> For configuring HPA/VPA objects and scaling **best practices**, use the > `gke-workload-scaling` skill instead. This skill focuses on **failure > diagnosis**.

## Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you MUST read [Usage Attribution](references/attribution.md). For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix `gcloud` with `CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-workload-scaling-troubleshooting)"` and append `agent-skills/1.0 (skill:gke-workload-scaling-troubleshooting)` to `User-Agent` headers.

## 🔍 Diagnosis & Resolution Workflow

### Step 0: Non-Interactive Context Discovery & Dry-Run Fallback

1. **Parameter Extraction**: Extract required context (`project_id`, `cluster_name`, `cluster_location`, `hpa_name`, `workload_name`, `workload_namespace`) non-interactively from the user prompt, active `SETTINGS.md`, or environment defaults:

- Default `workload_namespace` to `default` if omitted. - Infer missing cluster parameters from the active environment (`kubectl config current-context` or `gcloud config get-value project`).

2. **Cluster Credentials & Fallback Mode**:

- Attempt credential fetch: `gcloud container clusters get-credentials {cluster_name} --location {cluster_location} --project {project_id}`. - **Fallback / Dry-Run Mode**: If the cluster is unreachable, non-existent, or live command execution fails (such as in sandboxed evaluations, dry-run mode, or offline analysis): - Limit retry attempts to avoid resource exhaustion and context overflow. - Immediately present the exact `kubectl` / `gcloud` diagnostic commands for the human operator to run. - Synthesize the root-cause analysis and output the proposed GitOps correction based on the reported symptoms.

--------------------------------------------------------------------------------

### Step 1: Inspect the HPA and Classify the Symptom

Start every investigation with `kubectl describe hpa`, then route to the matching branch. The three key sections are **`Metrics`** (an `<unknown>` value means the HPA hasn't fetched the metric or the pipeline is broken), **`Conditions`** (`AbleToScale`, `ScalingActive`, `ScalingLimited` — a `False` status marks a failure), and **`Events`** (specific reasons such as `FailedGetScale` or `FailedGetResourceMetric`).

**Diagnostic Commands:**

```bash kubectl describe hpa {hpa_name} -n {workload_namespace} kubectl get hpa {hpa_name} -n {workload_namespace} -o yaml ```

For historical events, query Cloud Logging (the HPA events survive after the live `Events` list rolls over):

``` resource.type="k8s_cluster" resource.labels.cluster_name="{cluster_name}" resource.labels.location="{cluster_location}" logName="projects/{project_id}/logs/events" jsonPayload.involvedObject.kind="HorizontalPodAutoscaler" ```

Route by signal:

- **`FailedGetScale`, `FailedComputeMetricsReplicas`, `Error 400 ... label is not allowed`, or fluctuating replicas from competing HPAs** → **Branch A** (Configuration Errors). - **`FailedGetResourceMetric`, `unable to fetch pod metrics`, or `multiple services selecting the same target`** → **Branch B** (Workload & Service Errors). - **`<unknown>` custom/external metric, `FailedGetExternalMetric` / `FailedGetCustomMetric`, or `no known available metric versions found`** → **Branch C** (Metrics API & Data Availability). - **Conditions all `True` / no errors but the workload won't scale up or down** → **Branch D** (Healthy but Unexpected Scaling). - **Workload configured with `minReplicas: 0` won't scale to or from zero** → **Branch E** (Scale To / From Zero). - **Correct HPA but slow reaction on a cluster with many HPA objects** → **Branch F** (Slow Recalculation on Large Clusters).

--------------------------------------------------------------------------------

### Step 2: Resolution — Route to the Matching Branch

Based on the signal you classified in Step 1, jump to **one** of the mutually-exclusive branches below (A–F). These are alternatives — you do **not** run them in sequence. After applying the branch's fix, go to Step 3 to present it as a reviewable GitOps change.

#### Branch A: HorizontalPodAutoscaler Configuration Errors

- **`FailedGetScale` — `unable to get the target's current scale: ... "TARGET" not found`**: the `scaleTargetRef` doesn't resolve to an existing scalable workload.

- Verify the `scaleTargetRef` `name`, `kind`, and `apiVersion` exactly match the target workload's metadata. - Confirm the target workload exists **in the same namespace** as the HPA (a missing `-n` puts objects in `default`, causing a mismatch). - The target must be a scalable kind (Deployment, StatefulSet, ReplicaSet) — you **cannot** autoscale a DaemonSet.

- **`FailedComputeMetricsReplicas` — `invalid metrics (1 invalid out of 1)`**: the metric `type` and `target` don't match.

- If `type: Utilization`, the target must be `averageUtilization`. - If `type: AverageValue`, the target must be `averageValue`.

- **`unable to fetch metrics from external metrics API: googleapi: Error 400: Metric label: 'LABEL' is not allowed`**: an invalid key in `metric.selector.matchLabels`.

- Remove or correct the disallowed label; find valid filterable labels in the Cloud Monitoring metric documentation.

- **Replica count fluctuates / contradictory `SuccessfulRescale` events from different HPAs**: more than one HPA targets the same workload via `spec.scaleTargetRef`, and they compete. There is no dedicated condition for this — confirm with `kubectl get hpa -n {workload_namespace} -o yaml` and look for duplicate `scaleTargetRef` values.

- Consolidate all metrics into **one** HPA object (it takes the highest of its `spec.metrics`) and delete the duplicates.

--------------------------------------------------------------------------------

#### Branch B: Workload & Service Errors

- **`ScalingActive: False`, reason `FailedGetResourceMetric`, message `unable to compute the replica count`** (or a persistent `unable to fetch pod metrics`): the HPA computes utilization as a percentage of the container **resource request**, but at least one container in the Pod is missing a `resources.requests` entry for the scaled resource (`cpu` or `memory`).

- Add `resources.requests` for the scaled resource to **every** container in the Pod spec (including sidecars). A brief `unable to fetch pod metrics` right after the metrics server starts is normal and self-heals.

- **`multiple services selecting the same target of HPA_NAME: SERVICE`**: traffic-based autoscaling requires a **one-to-one** Service↔workload relationship, but more than one Service's selector matches the workload's Pods.

- Make the intended Service's selector unique (add a distinct label to the workload and to that one Service), or tighten the other Services' selectors so they no longer match the workload's Pods.

--------------------------------------------------------------------------------

#### Branch C: Metrics API & Data Availability (Custom / External Metrics)

The custom/external pipeline is: HPA controller → Kubernetes metrics API server → metrics adapter (for example `custom-metrics-stackdriver-adapter`) → metric source (Cloud Monitoring / Prometheus). Symptoms are `<unknown>` metric values or `FailedGetExternalMetric` / `FailedGetCustomMetric` events.

1. **Is the adapter registered and available?**

```bash kubectl get apiservice | grep -E 'NAME|metrics.k8s.io' ```

Expect `v1beta1.custom.metrics.k8s.io` and/or `v1beta1.external.metrics.k8s.io` with `AVAILABLE: True`. If `False`/missing, the adapter is crashed or misconfigured — inspect its Pod logs in the `custom-metrics` or `kube-system` namespace for permission, connectivity, or "metric not found" errors.

2. **Query the metrics API directly** (bypasses the HPA to test the whole pipeline; `jq` optional):

```bash kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/{workload_namespace}/{metric_name}" | jq . kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/{workload_namespace}/pods/*/{metric_name}" | jq . ```

3. **Interpret the result:**

- **Valid JSON with a value** → the pipeline works; the fault is in the HPA manifest (metric-name typo or wrong `matchLabels`). - **`Error from server (Service Unavailable)`** → network isolation is blocking the control plane from reaching the adapter. Add the adapter's `targetPort` to the **control-plane firewall rule** (in addition to the existing `tcp:443` and `tcp:10250`). Identify the rule with `gcloud compute firewall-rules list --filter="name~gke-{cluster_name}-[0-9a-z]*-master"`, and also confirm no NetworkPolicy blocks ingress to the adapter Pods. - **Empty list `[]`** → the adapter runs but can't retrieve the metric. Inspect the adapter Pod logs, and confirm in Metrics Explorer that the metric actually exists in the source with the expected name and labels.

- **`unable to fetch metrics from custom metrics API: no known available metric versions found`**: a communication breakdown (control plane briefly unavailable during an upgrade/repair, or the adapter Pods are unhealthy or not registered) — **not** a problem at the metric source. Check control-plane health/notifications, confirm the adapter Pods are `Running` with no restarts (`kubectl get pods -n custom-metrics,kube-system -o wide`), and re-verify the APIServices are `AVAILABLE: True`. Often transient.

- **`googleapi: Error 400: The supplied filter ... will not return any time series`**: the query is valid but no data matched (different from a value of `0`) — the application wasn't writing the metric during the window. Verify the metric name/labels match what the app emits, confirm the app had permission and was active, and check the app logs for metric-emission errors.

--------------------------------------------------------------------------------

#### Branch D: Healthy but Unexpected Scaling Behavior

The HPA's conditions are `True` and it shows no errors, but scaling doesn't happen as expected.

- **Won't scale up** — check, in order: - **Replica limits**: `currentReplicas` is already at `minReplicas` / `maxReplicas` (see the `ScalingLimited` condition); adjust the bounds. - **Tolerance window**: Kubernetes ignores changes while the current/target ratio stays within `0.9`–`1.1` (default **10% tolerance**). Example: target 85% CPU, current 93% → ratio ≈ 1.094 < 1.1, so no scale-up. Wait for the metric to move outside the band, or configure a different tolerance. - **Unready Pods**: `Pending`/not-`Ready` Pods are excluded from the calculation — resolve the underlying scheduling/probe issue. - **Sync delay**: a 15–30s delay between threshold crossing and action is normal.

- **Won't scale down** — check, in order: - **Multiple metrics**: the HPA uses the metric demanding the **most** replicas, so it won't scale down unless **all** metrics agree. - **Unavailable metric halts scale-down**: if any metric goes `<unknown>` the HPA conservatively refuses to scale down. Common with **rate-based** custom metrics that stop reporting at zero traffic. Prefer **gauge** metrics (for example `num_undelivered_messages`) or make the source publish `0` during inactivity rather than sending no data. - **Scale-down stabilization window**: the default `behavior.scaleDown.stabilizationWindowSeconds` is **300s (5 min)**. Lower it if scale-down must be faster.

--------------------------------------------------------------------------------

#### Branch E: Scale To / From Zero (GKE 1.37+)

Scaling a workload to and from zero replicas with HPA (`minReplicas: 0`) is supported on **GKE 1.37 or later**.

- **Won't scale to zero**: - `minReplicas: 0` must be set. - The HPA **cannot** scale to zero using only `Resource` (CPU/memory) metrics — configure at least one `External` or `Object` metric. - With multiple metrics, all must evaluate to zero. - GKE waits the 5-minute scale-down stabilization window at zero demand before going from 1→0. - Any metric showing `<unknown>` pauses scale-down (see Branch D).

- **Won't scale up from zero**: run `kubectl describe hpa {hpa_name}` and check: - **Metrics**: the value must be `> 0` and not `<unknown>`; if missing, confirm the external source (for example Pub/Sub) is publishing and Cloud Monitoring is receiving. - **Conditions**: a healthy idle state shows `AbleToScale: True`, `ScalingActive: True`, `ScaledToZero: True`. If `ScalingActive: False` with reason `ScalingDisabledExternalScaleToZero` or `ScalingDisabledReplicaCountZero`, the workload was **manually** scaled to zero (for example `kubectl scale`), which pauses autoscaling. Resume it by scaling the Deployment back to `--replicas=1`. - Cold-start latency from node provisioning can add delay; Capacity Buffers keep standby capacity ready.

--------------------------------------------------------------------------------

#### Branch F: Slow HPA Recalculation on Large Clusters

If HPAs are correct but react slowly, the cluster may exceed the HPA object count the standard controller keeps within a 15-second recalculation period.

- Standard controller: within 15s for up to **300 HPA objects** (GKE 1.22+). - **Performance HPA profile**: within 15s for up to **1,000 HPA objects** (GKE 1.31+) or **5,000 HPA objects** (GKE 1.33+, where it is enabled by default on eligible clusters).

Enable it on an eligible cluster (this is a cluster mutation — present it for the operator to run, don't execute it):

```bash gcloud container clusters update {cluster_name} \ --location {cluster_location} --project {project_id} \ --hpa-profile=performance ```

Scaling on many metrics per HPA and slow (>~50 ms) custom-metric adapters also lengthen the recalculation period. For visibility into scaling decisions, enable HPA event logging and review the structured HPA decision logs.

--------------------------------------------------------------------------------

### Step 3: Propose the GitOps Correction

Enforce the read-only diagnostics boundary: **do not** apply live mutations. This includes cluster/manifest changes (`kubectl edit`, `kubectl patch`, `kubectl apply`, `kubectl scale`, `kubectl delete`) **and** Google Cloud / gcloud changes (cluster updates such as `--hpa-profile`, and `gcloud compute firewall-rules` updates). Instead, present the corrected HorizontalPodAutoscaler, PodSpec (`resources.requests`), Service selector, workload autoscaling configuration, or the `gcloud` command as a reviewable patch/command to be applied through the user's GitOps pipeline (for example Config Sync, Argo CD, or Flux) or by an authorized operator.

## References

- [Troubleshoot horizontal Pod autoscaling in GKE](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/horizontal-pod-autoscaling.md.txt) - [Horizontal Pod autoscaling (concepts, Performance HPA profile & limits)](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/horizontalpodautoscaler.md.txt) - [Configure horizontal Pod autoscaling (hpa-profile)](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/horizontal-pod-autoscaling.md.txt) - [View horizontal Pod autoscaling events](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/view-horizontalpodautoscaling-events.md.txt) - [Scale to and from zero using HPA](https://docs.cloud.google.com/kubernetes-engine/docs/tutorials/scale-to-from-zero-hpa.md.txt) - [Kubernetes HorizontalPodAutoscaler (tolerance & scaling policies)](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)

Intended uses

  • Use Gke Workload Scaling Troubleshooting when this documented workflow matches the task.

Implementation notes

  • Review the source instructions and adapt inputs before running the workflow.

Try it, then tell us how it went

A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.