Skip to main content
Agent skill

Gke Alert Configuration

Configures alerting policies in Terraform for Google Kubernetes Engine (GKE) clusters, workloads, and services using PromQL and Google Cloud Managed Service for Prometheus. Use when writing, analyzing, validating, or deploying Terraform alerting policies to monitor GKE service latency, traffic, error rates using Multi-Window Multi-Burn-Rate SLO alerts, memory saturation, and cluster health such as CrashLoopBackOff and Node NotReady conditions. Don't use for non-GKE compute runtimes such as standalone Compute Engine VMs or standalone Cloud Run services without GKE.

Get started

Get the complete skill folder, including its scripts and reference files.

When to use this skill

Use Gke Alert Configuration when an AI agent needs a reusable SKILL.md workflow for this job: Configures alerting policies in Terraform for Google Kubernetes Engine (GKE) clusters, workloads, and services using PromQL and Google Cloud Managed Service for Prometheus. Use when writing, analyzing, validating, or deploying Terraform alerting policies to monitor GKE service latency, traffic, error rates using Multi-Window Multi-Burn-Rate SLO alerts, memory saturation, and cluster health such as CrashLoopBackOff and Node NotReady conditions. Don't use for non-GKE compute runtimes such as standalone Compute Engine VMs or standalone Cloud Run services without GKE.

When not to use it

Skip Gke Alert Configuration when the task is outside the coding category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.

How to install

Where will you use it?

Install scope
Claude Code folder setup
mkdir -p ~/.claude/skills/gke-alert-configuration
# Save SKILL.md to ~/.claude/skills/gke-alert-configuration/SKILL.md
  1. Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/gke-alert-configuration/SKILL.md. For a shared project, commit the folder at .claude/skills/gke-alert-configuration/ instead.
  2. Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
  3. For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.

Full install guide for Claude, Cursor, and Codex →

Try a starter prompt

After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.

Starter prompt
Use the Gke Alert Configuration skill for this task: Use Gke Alert Configuration when this documented workflow matches the task. Confirm any required inputs or files before making changes.

What supports this recommendation?

71/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.

Documentation
77/100
Practical detail
68/100
Source evidence
63/100
Source signals
72/100
Model compatibility evidence
  • claudeuntested

    No model-specific signal or recorded compatibility test was found.

  • chatgptuntested

    No model-specific signal or recorded compatibility test was found.

  • geminiuntested

    No model-specific signal or recorded compatibility test was found.

  • copilotuntested

    No model-specific signal or recorded compatibility test was found.

  • llamauntested

    No model-specific signal or recorded compatibility test was found.

  • perplexityuntested

    No model-specific signal or recorded compatibility test was found.

  • mistraluntested

    No model-specific signal or recorded compatibility test was found.

  • grokuntested

    No model-specific signal or recorded compatibility test was found.

Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.

What this skill does

# GKE Alert Configuration

This skill provides guidelines and best practices for creating robust, high-signal alerting policies for Google Kubernetes Engine workloads using Google Cloud Managed Service for Prometheus and Terraform. It ensures comprehensive coverage of the **4 Golden Signals** and key cluster health metrics while minimizing alert noise.

--------------------------------------------------------------------------------

## Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you MUST read [Usage Attribution](references/attribution.md). For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix `gcloud` with `CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-alert-configuration)"` and append `agent-skills/1.0 (skill:gke-alert-configuration)` to `User-Agent` headers.

## Critical Rules

* **Negative Triggers and Scope Redirection for Non-GKE Standalone Runtimes**: * This skill is strictly scoped to Google Kubernetes Engine (GKE) workloads, clusters, and services using PromQL and Google Cloud Managed Service for Prometheus. * **Do not use for non-GKE compute runtimes**, such as standalone Compute Engine virtual machines or standalone Cloud Run services without GKE. * **STOP AND RESPOND DIRECTLY (Do Not Edit Files)**: When the user requests alert configuration for non-GKE compute infrastructure: 1. **Do not write, create, edit, or validate any Terraform files on disk**. 2. **Immediately stop and respond directly to the user in chat**: * **Explicitly Clarify Out-of-Scope**: State clearly that standalone Compute Engine virtual machine monitoring or standalone Cloud Run monitoring is out of scope for this GKE-specific PromQL alerting skill, which is designed specifically for GKE workloads using Google Cloud Managed Service for Prometheus and PromQL. * **Do Not Generate GKE PromQL Alerts**: Do not create or generate Kubernetes PromQL alert policies or fabricate Kubernetes container, pod, or node resources for non-GKE infrastructure. * **Redirect the User**: Guide and redirect the user to standard Google Cloud Monitoring metrics, such as `compute.googleapis.com/instance/cpu/utilization` or `run.googleapis.com/request_latencies`, using standard `google_monitoring_alert_policy` with `condition_threshold` or MQL, or recommend the relevant specialized Cloud observability skill. * **Mandatory `kube-state-metrics` (KSM) Cost Guardrail**: * Deploying open-source `kube-state-metrics` in Google Cloud Managed Service for Prometheus incurs billable metric ingestion costs. * **STOP AND ASK PERMISSION FIRST (Do Not Edit Files)**: When a requested alert rule relies on **Tier 2 KSM metrics** (such as `kube_cronjob_*`, `kube_pod_status_phase`, `kube_persistentvolume_*`, `kube_deployment_*`, `kube_statefulset_*`, `kube_job_*`, or `kube_daemonset_*`), **do not write, create, edit, or validate any Terraform files or generate alert policies before obtaining user approval**. * Instead, you **must immediately stop and respond directly to the user** to: 1. **Alert the user** that the requested alert requires `kube-state-metrics`. 2. **Explain the cost impact**: Detail that `kube-state-metrics` incurs billable sample ingestion costs in Google Cloud Managed Service for Prometheus. 3. **Ask for explicit permission**: Ask the user for explicit permission before assuming, enabling, or generating KSM-dependent alert configurations. 4. **Recommend filtering or allowlisting**: Suggest and recommend filtering or allowlisting only the specific required metrics, such as using a `PodMonitoring` resource with `metricRelabeling` (`action: keep`) or KSM `--metric-allowlist` to minimize ingestion costs. Provide a concrete allowlist example. * **Always prefer Non-KSM Native Alternatives** (Tier 1 cAdvisor or native GKE metrics documented in [metrics_and_alerts_catalog.md](references/metrics_and_alerts_catalog.md)) whenever possible, such as using `container_memory_working_set_bytes` and `container_spec_memory_limit_bytes` instead of `kube_pod_container_resource_limits`. * **Explicit Tier and Cost Surcharge Identification in Response**: In every response where you generate or recommend an alerting policy, you **must explicitly state its classification tier and cost impact**: * **Tier 1 native or standard metric** (GKE built-in metrics, cAdvisor `container_*`, kubelet volume stats, kubelet node conditions, and control-plane metrics; see [metrics_and_alerts_catalog.md](references/metrics_and_alerts_catalog.md)): State that it is a **Tier 1 native or standard metric with zero KSM cost surcharge**. * **Tier 2 KSM metric**: State that it is a **Tier 2 KSM-dependent metric** and follow the permission and allowlisting guardrail above. *(Tip: Generally, metrics with the `kube_` prefix that represent resource state or metadata belong to Tier 2).* * **Plan-Validate-Execute Loop for Approved File Edits**: When modifying, adding, or merging approved Terraform files on disk in a workspace, follow the three-phase workflow: 1. **Plan**: Draft a structured change plan (`changes.json`) containing proposed policy resource names, PromQL expressions, grouping labels, and durations. 2. **Validate**: Run the pre-edit validation script (`python3 scripts/validate_config.py --plan changes.json`) to verify PromQL grammar, lookback windows, duration rules, and ensure no duplicate signals exist. 3. **Execute**: After the plan passes validation, apply or merge changes in-place into the target Terraform configuration (`alerts.tf`). 4. *Note*: When answering questions or providing Terraform snippets directly in chat where no disk modification is requested, output the complete, valid Terraform HCL block in your response. * **Configure the 4 Golden Signals and Cluster Health**: Always ensure the target Kubernetes workload or service has the following alerting coverage: 1. **Latency** (P95 response time) 2. **Errors** (Multi-Window Multi-Burn-Rate SLO alerts, such as Fast Burn 1 hour / 5 minutes with factor 14.4, Slow Burn 6 hours / 30 minutes with factor 6.0; do not use simple static ratios) 3. **Traffic** (Sudden drop or complete metric disappearance using `absent()` or `default 0` syntax, or overload spikes) 4. **Saturation (Memory Limit Utilization Only)**: When describing or configuring alert policies for a cluster or project, include ONLY **Memory Saturation** (`container_memory_working_set_bytes` / `container_spec_memory_limit_bytes`). Do **NOT** include CPU saturation alerts or list `container_cpu_usage_seconds_total` as an alert metric because CPU is compressible and throttled by CFS quotas rather than causing uncompressible fatal termination (OOM). 5. **Cluster Health** (Pod CrashLooping, Node NotReady) * **PromQL Only (Managed Prometheus)**: You must use `condition_prometheus_query_language` with PromQL. Do **NOT** use MQL or standard `condition_threshold` unless explicitly requested. Google Cloud Managed Service for Prometheus is the standard telemetry ingestion path for GKE. * **Terraform Only**: Write the generated observability configuration ONLY as Terraform (`.tf`) files, such as `alerts.tf` and `variables.tf`. * **Dynamic Multi-Resource Alerting (No Hardcoding)**: You must not hardcode specific pod names, node names, or service names in alerting conditions unless explicitly requested. Alerting policies must be written to cover resources dynamically: * Always use grouping aggregations (`by (cluster, namespace, service, pod, container)`) instead of filtering to a single instance. This allows a single alert policy to dynamically track each service or pod separately. * Always declare and use Terraform variables for `project_id`, `cluster_name`, and `namespace` (`var.project_id`, `var.cluster_name`, `var.namespace`) to make the configuration reusable across environments. Always define these variables in `variables.tf` (or within the configuration) and reference all three in policies or PromQL label matchers. * **No Redundant Duration Windows on Lookbacks**: * When PromQL expressions already use an aggregated lookback window (such as `increase(...[15m]) > 3` or multi-window SLO burn rates), the query time window already smooths out transient spikes. * Adding a Terraform duration on top of a PromQL lookback window increases the Mean Time to Detect (MTTD) without providing additional smoothing benefits. * In these cases, set Terraform `duration = "0s"` (or `"60s"`). Do not enforce `duration = "300s"` on top of `[15m]`, which delays critical crashloop alerts by up to 20 minutes total (15 minutes + 5 minutes). * Use `duration = "300s"` only on instantaneous gauge conditions, such as `kube_node_status_condition == 0`. * **Use SLO Burn Rates Instead of Simple Ratios**: For error rate alerting, always generate Multi-Window Multi-Burn-Rate (MWMBR) SLO alerts (such as 14.4x burn rate over 1 hour and 5 minute windows for a 99% SLO) rather than simple error rate ratios (`rate(5xx)/rate(total) > 0.05`), which produce excessive false alarms on low traffic. * **Robust Traffic Drop Detection (`absent()` / `default 0`)**: When monitoring for traffic drops to zero, do not use `rate(...) == 0` alone because Prometheus time series disappear completely when no requests occur (evaluating to an empty vector rather than 0). Use `default 0` syntax, such as `sum(rate(...[5m])) default 0 == 0`, or `absent(...) == 1`. * **Notification Channels**: By default, never configure any notification channels without user input. If the user explicitly provides a notification channel, configure the alerts to use it. Otherwise, you must prompt the user in your response to ask if they would like to configure one. * **Consult GKE Metrics and Open-Source Alerts Catalog**: When designing or generating evaluation suites or alerting policies, consult [metrics_and_alerts_catalog.md](references/metrics_and_alerts_catalog.md) for public GKE metrics (`kubernetes.io/`) and open-source Kubernetes alerts (`awesome-prometheus-alerts`). * **Plain English Response**: You must include a plain English explanation for what the alerts do in your response. Explain what the alert measures, what the threshold represents, and what a trigger indicates. * **User Labels**: Include a `user_labels` block in all `google_monitoring_alert_policy` resources to track policies created by this skill:

```terraform user_labels = { created-with-google-skill = "gke-alert-configuration" } ```

--------------------------------------------------------------------------------

## Alerting Policy Structure in Terraform

Alerting policies must be defined using the `google_monitoring_alert_policy` resource with `condition_prometheus_query_language`. Always declare variables in `variables.tf` for `project_id`, `cluster_name`, and `namespace`.

```hcl # variables.tf variable "project_id" { type = string description = "Google Cloud Project ID" }

variable "cluster_name" { type = string description = "GKE Cluster Name" }

variable "namespace" { type = string description = "Target Kubernetes Namespace" default = "default" }

variable "slo_target" { type = number description = "SLO Target fraction (for example 0.99 for 99%)" default = 0.99 } ```

```hcl # alerts.tf # Example: Multi-Window Multi-Burn-Rate (MWMBR) SLO Alert (Fast Burn: 14.4x, 1h & 5m windows) resource "google_monitoring_alert_policy" "k8s_service_error_rate_slo" { project = var.project_id display_name = "[K8s] ${var.cluster_name} - Service Error Rate SLO Fast Burn" combiner = "OR"

conditions { display_name = "Error Budget Fast Burn (14.4x over 1h and 5m)" condition_prometheus_query_language { query = <<-EOT ( ( sum( rate( http_requests_total{ cluster="${var.cluster_name}", namespace="${var.namespace}", status=~"5.." }[5m] ) ) by (service, namespace, cluster) / sum( rate( http_requests_total{ cluster="${var.cluster_name}", namespace="${var.namespace}" }[5m] ) ) by (service, namespace, cluster) ) > (1 - ${var.slo_target}) * 14.4 ) and ( ( sum( rate( http_requests_total{ cluster="${var.cluster_name}", namespace="${var.namespace}", status=~"5.." }[1h] ) ) by (service, namespace, cluster) / sum( rate( http_requests_total{ cluster="${var.cluster_name}", namespace="${var.namespace}" }[1h] ) ) by (service, namespace, cluster) ) > (1 - ${var.slo_target}) * 14.4 ) EOT duration = "0s" } } } ```

--------------------------------------------------------------------------------

## Telemetry Metrics and PromQL Examples

For GKE metrics (`kubernetes.io/`), community open-source alerts (`awesome-prometheus-alerts`), KSM cost guardrails, and non-KSM native alternatives, you must read and follow:

* [metrics_and_alerts_catalog.md](references/metrics_and_alerts_catalog.md)

For specific PromQL queries corresponding to each of the Golden Signals, you must read and follow:

* [promql_queries.md](references/promql_queries.md)

For GKE cluster prerequisites, enabling Google Cloud Managed Service for Prometheus collection, configuring PodMonitoring custom scraping, and enabling control plane metrics collection (API Server, Controller Manager, Scheduler), you must read and follow:

* [gke_configuration_prerequisites.md](references/gke_configuration_prerequisites.md)

--------------------------------------------------------------------------------

## Tooling Scripts and Validation Loop

Use the `validate_config.py` script to validate change plans and Terraform configurations when working in a repository:

* **Pre-Edit Plan Validation**: Draft a `changes.json` plan specifying the proposed policies, queries, and durations, and validate it before editing: * Command: `python3 scripts/validate_config.py --plan changes.json` * **Post-Edit and Directory Validation**: Scan existing or modified Terraform files in a directory to ensure no duplicates or syntax errors exist: * Command: `python3 scripts/validate_config.py --directory [TARGET_TF_DIR] --cluster-var "${var.cluster_name}"` * Single file validation: `python3 scripts/validate_config.py --file [PATH_TO_TF_FILE]`

--------------------------------------------------------------------------------

## Technical Considerations and Gotchas

* **Lookback Windows versus Duration Buffers**: * Do not add large `duration = "300s"` buffers to alerts that already use aggregated lookback windows like `increase(...[15m])` or multi-window SLO rates. * The `[15m]` window in `increase(...[15m]) > 3` already smooths spikes. Adding `duration = "300s"` increases MTTD by forcing the restart count to remain above 3 for an extra 5 continuous minutes, delaying alerts by up to 20 minutes total. * Use `duration = "0s"` or `"60s"` when using lookback window functions. Reserve `duration = "300s"` for raw instantaneous gauge conditions, such as `kube_node_status_condition == 0`. * **Memory Saturation Only for Cluster Alerting**: * Do not configure CPU saturation alerts for cluster or workload monitoring. CPU is compressible (throttled by the CFS scheduler), while memory is uncompressible (triggers OOMKills). * Configure Memory Saturation using `container_memory_working_set_bytes` / `container_spec_memory_limit_bytes`. * **Missing Resource Limits Blind Spot (Mandatory Explanation)**: Saturation alerts that compare usage to limits (such as `container_spec_memory_limit_bytes`) will **fail to resolve** or return `NaN` if workloads do not have explicit Memory limits configured in their Kubernetes manifests. * **Mandatory Instruction**: Whenever you generate, discuss, or recommend any memory saturation alert comparing usage against limits (including non-KSM cAdvisor alternatives using `container_spec_memory_limit_bytes`), you **must explicitly explain and warn the user in your response** that container memory limits must be explicitly configured in the Kubernetes pod specs or manifests (`resources.limits.memory`) for the saturation query to resolve (and not return `NaN` or fail to resolve). * **Linear Disk Predictions (`predict_linear`)**: When forecasting volume exhaustion using `predict_linear(kubelet_volume_stats_available_bytes[6h:5m], 4 * 24 * 3600) < 0`, explain that `predict_linear` uses linear regression over the recent lookback window (for example, 6 hours) to project when available disk will drop below 0 (for example, within 4 days). Identify `kubelet_volume_stats_available_bytes` as a Tier 1 native kubelet metric with zero KSM surcharge. * **API Server Error and Client Metrics**: * `apiserver_request_total` and `rest_client_requests_total` are Tier 1 Control Plane metrics with zero KSM cost surcharge. Explain that `apiserver_request_total` monitors 5xx HTTP error rates across API server endpoints, while `rest_client_requests_total` monitors 4xx and 5xx requests sent by REST clients communicating with the API server. * **Traffic Disappearance Gotcha (`absent()` / `default 0`)**: * When traffic drops completely to zero, Prometheus and GMP stop emitting the `http_requests_total` time series. * `sum(rate(...[5m])) == 0` evaluates to an empty vector, preventing the alert from triggering. * Always use `sum(rate(...[5m])) default 0 == 0` or `absent(...) == 1` to reliably detect total traffic loss. * **CrashLooping versus Normal Restarts**: A container restarting occasionally might be normal, for example job completion or a minor rolling update. Alert on **frequent** restarts (such as more than 3 restarts in 15 minutes with `duration = "0s"`) using `kube_pod_container_status_restarts_total` rather than a single restart to avoid noise. * **Node Upgrades**: During GKE cluster upgrades, nodes are drained and restarted, which can trigger "Node NotReady" alerts. Warn the user that these alerts might fire during maintenance windows, or suggest configuring maintenance windows if supported.

--------------------------------------------------------------------------------

## Additional Resources

* [Google Cloud Managed Service for Prometheus Documentation](https://docs.cloud.google.com/stackdriver/docs/managed-prometheus.md.txt) * [GKE Observability and Monitoring Concepts](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/observability.md.txt) * [Google Cloud Alerting Policies in Terraform](https://docs.cloud.google.com/monitoring/alerts/terraform.md.txt) * [Google Cloud Monitoring Pricing](https://docs.cloud.google.com/monitoring/pricing.md.txt * [Google SRE Workbook: Alerting on SLOs](https://sre.google/workbook/alerting-on-slos/) * [Awesome Prometheus Alerts Repository](https://github.com/samber/awesome-prometheus-alerts)

Intended uses

  • Use Gke Alert Configuration when this documented workflow matches the task.

Implementation notes

  • Review the source instructions and adapt inputs before running the workflow.

Try it, then tell us how it went

A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.