cloudnative wiki

Google Kubernetes Engine (GKE)

9 min read

Google Kubernetes Engine (GKE) ☸️

Google Kubernetes Engine is Google Cloud’s managed Kubernetes service. As the original creators of Kubernetes, Google designed GKE with native cloud integrations: VPC-native networking (Alias IPs), Datapath V2 (kernel eBPF via Cilium), GKE Autopilot (fully managed nodes & pod-level billing), and Container-Native Load Balancing.

Architecture & Mental Model

GKE Architecture Overview

                   Google-Managed Control Plane (Regional HA)
                   ┌──────────────────────────────────────┐
                   │  etcd (Quorum)  │  kube-apiserver    │ (Across 3 Zones)
                   │  kube-scheduler │  controller-mgr    │ 99.95% SLA
                   └──────────────────┬───────────────────┘
                                      │
               ┌──────────────────────┴──────────────────────┐
               │ Private Peering / Konnectivity Tunnel        │
               ▼                                             ▼
    ┌─────────────────────────┐                   ┌─────────────────────────┐
    │  Worker Node: Zone-a    │                   │  Worker Node: Zone-b    │
    │  ┌───────────────────┐  │                   │  ┌───────────────────┐  │
    │  │ Pod 1 (10.101.0.2)│  │                   │  │ Pod 3 (10.101.1.2)│  │
    │  └───────────────────┘  │                   │  └───────────────────┘  │
    │  ┌───────────────────┐  │                   │  ┌───────────────────┐  │
    │  │ Pod 2 (10.101.0.3)│  │                   │  │ Pod 4 (10.101.1.3)│  │
    │  └───────────────────┘  │                   │  └───────────────────┘  │
    │  Datapath V2 (eBPF)     │                   │  Datapath V2 (eBPF)     │
    │  GKE Metadata Server    │                   │  GKE Metadata Server    │
    └──────────┬──────────────┘                   └──────────┬──────────────┘
               │                                             │
               └────────── Subnet Primary Range (Node IPs) ──┘
                         Subnet Secondary Range (Pod IPs)

Autopilot vs. Standard Mode

Architecture DimensionGKE StandardGKE Autopilot
Node ManagementCustomer manages node pools, instance types, OS image, and scalingFully automated: Google provisions, configures, and scales nodes
Billing ModelPay for the underlying VM instances + disk, regardless of pod utilizationPay only for pod resource requests (vCPU, memory, ephemeral storage)
Security PostureDefault settings can be weakened; customer handles CIS benchmarkHardened by default: Shielded nodes, Workload Identity, secure baseline
Node Access / SSHFull root SSH access to nodes enabledNode access restricted; no SSH (managed infrastructure)
Custom DaemonSetsUnrestricted custom DaemonSets permittedPermitted with strict resource request restrictions
Ideal ForCustom kernels, specialized hardware configs, legacy setupsProduction default for modern containerized cloud-native apps

GKE Deep Dive Engineering Modules 📚

Explore the exhaustive, battle-tested GKE architectural reference library:

1. Cluster Topologies & Operational Modes

2. Capacity & Advanced Autoscaling

3. Ingress, Networking & Service Discovery

4. Storage & Stateful Systems

5. AI/ML Accelerators & Batch Processing

6. Enterprise Security & Multi-Tenancy

7. Observability, Disaster Recovery & Incident Response

8. FinOps & Fleet Governance

Core Concepts

1. VPC-Native Clusters & Alias IPs

Every production GKE cluster should be created in VPC-native mode (using VPC Alias IPs):

  • No Overlay Encapsulation: Pods get real, routable IP addresses directly from the VPC subnet’s secondary IP range.
  • Direct Routability: Compute Engine VMs, Cloud SQL databases, and on-premises networks (via Cloud Interconnect) can route directly to Pod IPs without NAT.
  • Container-Native Load Balancing: Google Cloud Load Balancer routes traffic directly to Pod IPs via Zonal NEGs, completely bypassing kube-proxy iptables overhead.

2. Datapath V2 (eBPF / Cilium)

Datapath V2 is GKE’s modern networking dataplane built on open-source Cilium and Linux kernel eBPF:

  • Replaces kube-proxy: Routing, service load balancing, and network policies are executed in the Linux kernel via eBPF bytecode instead of sequential iptables evaluation.
  • Extreme Scale: Eliminates the CPU performance cliff encountered when clusters grow to thousands of Kubernetes Services.
  • Built-in Network Policy Enforcement: Enforces Kubernetes NetworkPolicy objects natively without installing third-party CNI plugins (like Calico).

3. Regional vs. Zonal Clusters

  • Zonal Cluster: Control plane runs in a single zone. If that zone experiences an outage or control plane upgrade, the Kubernetes API server is unavailable (existing pods continue running). SLA: 99.5%.
  • Regional Cluster: Control plane replicates across three zones in the region with automated multi-zone etcd quorum. SLA: 99.95%. Mandatory for all production environments.

4. Private GKE Clusters & Control Plane Access

In a private cluster:

  • Worker nodes have NO public IPs: Nodes reside exclusively in private subnets and reach the public internet via Cloud NAT.
  • Private Control Plane Endpoint: The API server communicates with nodes over a private peering network.
  • Master Authorized Networks: Restricts access to the Kubernetes API server endpoint to explicit corporate CIDR blocks or bastion jump hosts.

Production gcloud CLI Commands

1. Creating a Production GKE Autopilot Regional Cluster

gcloud container clusters create-auto prod-autopilot-cluster \
  --region=us-central1 \
  --network=prod-vpc \
  --subnetwork=prod-us-central1 \
  --cluster-secondary-range-name=gke-pods \
  --services-secondary-range-name=gke-services \
  --enable-private-nodes \
  --master-ipv4-cidr=172.16.0.0/28 \
  --enable-master-authorized-networks \
  --master-authorized-networks=203.0.113.50/32 \
  --release-channel=regular

2. Creating a Production GKE Standard Private Cluster with Datapath V2

gcloud container clusters create prod-standard-cluster \
  --region=us-central1 \
  --release-channel=regular \
  --network=prod-vpc \
  --subnetwork=prod-us-central1 \
  --cluster-secondary-range-name=gke-pods \
  --services-secondary-range-name=gke-services \
  --enable-ip-alias \
  --enable-dataplane-v2 \
  --workload-pool=my-prod-project.svc.id.goog \
  --enable-private-nodes \
  --master-ipv4-cidr=172.16.0.16/28 \
  --num-nodes=1 \
  --enable-autoscaling \
  --min-nodes=1 \
  --max-nodes=5 \
  --node-locations=us-central1-a,us-central1-b,us-central1-c \
  --shielded-secure-boot \
  --shielded-integrity-monitoring

3. Deploying Gateway API (Next-Gen Ingress)

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: prod-external-gateway
  namespace: production
spec:
  gatewayClassName: gke-l7-global-external-managed
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      tls:
        mode: Terminate
        certificateRefs:
          - name: prod-tls-cert
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-route
  namespace: production
spec:
  parentRefs:
    - name: prod-external-gateway
  hostnames:
    - "api.company.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /v1/users
      backendRefs:
        - name: user-service
          port: 8080

Quotas & Limits

Resource / ParameterLimitProduction Notes
Max nodes per cluster15,000 nodesStandard cluster scale limit
Max pods per cluster300,000 podsRequires careful CIDR capacity planning
Default max pods per node110 podsConfigurable down to 8–64 to conserve CIDR space
Control plane management fee72/mo)Free tier: 1 free cluster management fee per billing account
Master IPv4 CIDRMust be /2816 IPs used exclusively by Google control plane VMs

References

Pricing Examples

Scenario 1: GKE Standard Cluster (Sustained Workload)

  • 1 Regional Cluster (72.00 / month management fee).
  • 9 Worker Nodes across 3 zones using e2-standard-4 (4 vCPU, 16 GB RAM each).
  • Compute cost: 9 × ~873.00.
  • Storage: 9 × 100 GB pd-balanced boot disks (90.00).
  • Total Cost: 873 + 1,035 / month**.

Scenario 2: GKE Autopilot Cluster (Variable Microservices)

  • 1 Regional Autopilot Cluster (72.00 / month management fee, waived if first cluster on billing account).
  • Workload footprint: 100 microservice pods averaging 0.5 vCPU and 1 GB RAM requested per pod.
  • Total requested: 50 vCPUs, 100 GB RAM continuously.
  • Autopilot pricing (us-central1):
    • vCPU: 50 × 1,624.25
    • RAM: 100 × 359.34
  • Total Autopilot Cost: ~359.34 + 2,055.59 / month** (Zero node management overhead, zero idle node capacity wasted).

Nuggets & Gotchas

  1. The /28 Master CIDR Cannot Overlap Anything: When creating a private GKE cluster, the --master-ipv4-cidr requires a /28 block. This block is peered directly to your VPC. If this /28 overlaps with any existing subnet, peered VPC, or on-premises CIDR routed via VPN, cluster creation will fail after 20 minutes with a vague peering conflict error.
  2. Default Max Pods per Node Consumes /24 Secondary IP Space: By default, GKE allocates a /24 (256 addresses) from the Pod secondary CIDR for every single node, even if the node only runs 10 pods. For a 100-node cluster, this burns 25,600 Pod IPs. Use --default-max-pods-per-node=32 or 64 to dramatically reduce subnet consumption.
  3. Autopilot Resource Request Enforcement: In GKE Autopilot, Google bills you based on Resource Requests, not limits. Furthermore, if you specify limits without requests, Autopilot automatically sets requests = limits. If an unconfigured pod has a memory limit of 8 GiB, Autopilot charges you for 8 GiB from the instant it schedules!
  4. Maintenance Windows & Automated Upgrades: GKE automatically upgrades control planes and nodes to track the selected Release Channel. Without a configured Maintenance Window and Exclusion Window, upgrades can trigger during peak production hours, causing rolling pod restarts. Always configure an explicit maintenance window (e.g., Saturday 02:00-06:00 UTC).
  5. Kubelet Eviction on Boot Disk Exhaustion: In GKE Standard, container logs written to stdout/stderr are stored on the node’s root boot disk under /var/log/pods. If an application enters a crash loop and emits gigabytes of logs per minute, the node’s boot disk will fill to 85%, triggering DiskPressure and evicting all non-daemonset pods from that node.

Across the wiki

22 items under this folder.