Gke Inference
Deploys and optimizes AI/ML inference workloads on GKE, using GPUs, TPUs, and model servers. Use when deploying GKE inference servers, configuring GKE GPU resources for inference, or deploying LLMs on GKE. Don't use for migrating existing AI workloads to GKE (use google-cloud-solution-guided-gke-ai-migration), GKE RAG with Cloud SQL/AlloyDB (use google-cloud-solution-rag-enterprise-search-gke-sqldb), or batch/HPC (use gke-batch-hpc).
Get the complete skill folder, including its scripts and reference files.
When to use this skill
Use Gke Inference when an AI agent needs a reusable SKILL.md workflow for this job: Deploys and optimizes AI/ML inference workloads on GKE, using GPUs, TPUs, and model servers. Use when deploying GKE inference servers, configuring GKE GPU resources for inference, or deploying LLMs on GKE. Don't use for migrating existing AI workloads to GKE (use google-cloud-solution-guided-gke-ai-migration), GKE RAG with Cloud SQL/AlloyDB (use google-cloud-solution-rag-enterprise-search-gke-sqldb), or batch/HPC (use gke-batch-hpc).
When not to use it
Skip Gke Inference when the task is outside the creativity category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.
How to install
Where will you use it?
mkdir -p ~/.claude/skills/gke-inference # Save SKILL.md to ~/.claude/skills/gke-inference/SKILL.md
- Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/gke-inference/SKILL.md. For a shared project, commit the folder at .claude/skills/gke-inference/ instead.
- Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
- For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.
Full install guide for Claude, Cursor, and Codex →
Try a starter prompt
After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.
Use the Gke Inference skill for this task: Deploy an AI model (Llama, Gemma, Mistral, etc.) to GKE. Confirm any required inputs or files before making changes.
What supports this recommendation?
87/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.
- Documentation
- 98/100
- Practical detail
- 98/100
- Source evidence
- 63/100
- Source signals
- 70/100
Model compatibility evidence
- claudeinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- chatgptinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- geminiinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- copilotinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- llamainferred
The skill text mentions Llama or a closely associated term. This is keyword-derived compatibility; no hands-on model test is recorded.
- perplexityinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
- mistralinferred
The skill text mentions Mistral or a closely associated term. This is keyword-derived compatibility; no hands-on model test is recorded.
- grokinferred
The skill uses model-agnostic prompt or LLM terminology. Portability is inferred and has not been verified on this model.
Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.
What this skill does
# GKE AI/ML Inference
> **Routing Note:** For migrating existing AI workloads to GKE, open `google-cloud-solution-guided-gke-ai-migration/SKILL.md`. For GKE RAG with Cloud SQL or AlloyDB (`pgvector`), open `google-cloud-solution-rag-enterprise-search-gke-sqldb/SKILL.md`.
This reference covers deploying AI/ML inference workloads on GKE using Google's Inference Quickstart (GIQ) and best practices for LLM serving.
> **MCP Tools:** `apply_k8s_manifest`, `get_k8s_resource`, `get_k8s_logs`, > `get_k8s_rollout_status`, `describe_k8s_resource`, `list_k8s_events`. > **CLI-only:** `gcloud container ai profiles *`
## Attribution
**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you MUST read [Usage Attribution](references/attribution.md). For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix `gcloud` with `CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-inference)"` and append `agent-skills/1.0 (skill:gke-inference)` to `User-Agent` headers.
## When to Use
- Deploy an AI model (Llama, Gemma, Mistral, etc.) to GKE - Generate optimized Kubernetes manifests for inference - Select GPU/TPU accelerators for model serving - Configure autoscaling for LLM inference
## Prerequisites
- A golden path GKE Autopilot cluster (GPU workloads are supported via ComputeClasses and NAP) - `gcloud` CLI authenticated - Sufficient GPU/TPU quota in the target region
## Workflow
### 1. Discovery: Find Models and Hardware
```bash # List all supported models gcloud container ai profiles models list --quiet
# Find valid accelerator/server combinations for a model gcloud container ai profiles list --model=<MODEL_NAME> --quiet
# Example: what can run Gemma 2 9B? gcloud container ai profiles list --model=gemma-2-9b-it --quiet ```
### 2. Generate Manifest
```bash gcloud container ai profiles manifests create \ --model=<MODEL_NAME> \ --model-server=<SERVER> \ --accelerator-type=<ACCELERATOR> \ --target-ntpot-milliseconds=<NTPOT> --quiet > inference.yaml ```
**Parameters:**
- `--model`: Model ID (e.g., `gemma-2-9b-it`, `llama-3-8b`) - `--model-server`: Inference server (`vllm`, `tgi`, `triton`, `tensorrt-llm`) - `--accelerator-type`: GPU/TPU type (`nvidia-l4`, `nvidia-tesla-a100`, `nvidia-h100-80gb`) - `--target-ntpot-milliseconds`: Target Normalized Time Per Output Token (optional, for latency optimization)
**Example:**
```bash gcloud container ai profiles manifests create \ --model=gemma-2-9b-it \ --model-server=vllm \ --accelerator-type=nvidia-l4 \ --target-ntpot-milliseconds=50 --quiet > inference.yaml ```
### 3. Review and Deploy
```bash # Review for placeholders (HF tokens, PVCs) cat inference.yaml
# Deploy kubectl apply -f inference.yaml
# Monitor kubectl get pods -w kubectl logs -f <POD_NAME> ```
> Some models require Hugging Face tokens. Create a Kubernetes Secret and > reference it in the manifest.
## GPU ComputeClass for Inference
For Autopilot clusters, create a ComputeClass to target GPU nodes:
```yaml apiVersion: cloud.google.com/v1 kind: ComputeClass metadata: name: l4-inference spec: priorities: - machineFamily: g2 gpu: type: nvidia-l4 count: 1 minCores: 4 minMemoryGb: 16 ```
## Accelerator Selection Guide
| Accelerator | Best For | Memory | Relative Cost | | ------------------- | ------------------------ | ----------- | ------------- | | NVIDIA T4 | Budget inference, | 16 GB | Lowest | : : lightweight legacy : : : : : models : : : | NVIDIA L4 (G2) | Small-medium model | 24 GB | Low | : : inference, video, : : : : : graphics : : : | NVIDIA RTX PRO 6000 | Multimodal AI, | 96 GB | Medium | : (G4) : high-fidelity 3D, : : : : : fine-tuning : : : | Cloud TPU v5e | Cost-effective | Varies | Medium | : : transformer inference : : : | Cloud TPU v5p | High-performance | Varies | High | : : training : : : | Cloud TPU v6e | High-efficiency next-gen | 32 GB/chip | Medium-High | : (Trillium) : training & serving : : : | Cloud TPU v7x | Ultra-scale inference & | 192 GB/chip | High | : (Ironwood) : agentic workflows : : : | NVIDIA A100 | Large model inference, | 40/80 GB | High | : : enterprise ML : : : | NVIDIA H100 / H200 | Frontier model training, | 80/141 GB | Highest | : : high throughput : : : | NVIDIA B200 (A4) | Blackwell-scale | 192 GB | Highest | : : training, FP4 precision : : : | NVIDIA GB200 (A4X) | Rack-scale AI (Grace | Massive | Highest | : : Blackwell Superchip) : : :
## Autoscaling LLM Inference
### GPU-based autoscaling
Use custom metrics for GPU utilization:
```yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: llm-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: llm-server minReplicas: 1 maxReplicas: 10 metrics: - type: Pods pods: metric: name: gpu_duty_cycle target: type: AverageValue averageValue: "80" ```
### Best practices for inference autoscaling
1. **Use DCGM metrics**: Golden path enables DCGM monitoring for GPU utilization metrics 2. **Set appropriate minReplicas**: At least 1 for always-on serving; 0 for batch/on-demand 3. **Tune scale-down delay**: LLM model loading is slow; use longer stabilization windows 4. **Consider queue depth**: Scale on pending requests rather than pure GPU utilization for latency-sensitive workloads
## Optimization Tips
- **Quantization**: Use quantized models (GPTQ, AWQ) to reduce GPU memory and increase throughput - **Batching**: Configure model server batch size for throughput vs latency trade-off - **Tensor parallelism**: Split large models across multiple GPUs within a node - **KV cache optimization**: Tune `--gpu-memory-utilization` in vLLM for KV cache allocation
## Troubleshooting
| Issue | Cause | Fix | | ------------------ | ------------------------ | --------------------------- | | Invalid | Unsupported tuple | Re-run `gcloud container ai | : model/accelerator : : profiles list : : combination : : --model=<MODEL>` : | GPU quota exceeded | Regional quota limit | Request quota increase or | : : : try a different region : | OOM on GPU | Model too large for | Use larger GPU, enable | : : accelerator : quantization, or use tensor : : : : parallelism : | Slow cold start | Large model loading from | Use local SSD for model | : : registry : caching; pre-pull images :
Intended uses
- Deploy an AI model (Llama, Gemma, Mistral, etc.) to GKE
- Generate optimized Kubernetes manifests for inference
- Select GPU/TPU accelerators for model serving
- Configure autoscaling for LLM inference
Implementation notes
- Use DCGM metrics: Golden path enables DCGM monitoring for GPU
- Set appropriate minReplicas: At least 1 for always-on serving; 0 for
- Tune scale-down delay: LLM model loading is slow; use longer
- Consider queue depth: Scale on pending requests rather than pure GPU
Try it, then tell us how it went
A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.