GKE Networking Deep Dive — Datapath V2, Alias IPs & Gateway API ☸️🌐

Networking in Google Kubernetes Engine (GKE) is deeply integrated into Google Cloud’s software-defined Andromeda network fabric. Unlike traditional on-prem Kubernetes setups that rely on overlay encapsulation (VXLAN/Geneve) and sequential iptables routing, GKE provides VPC-native routing with Alias IPs, kernel-level eBPF datapaths (Cilium Datapath V2), and Container-Native Load Balancing via Zonal NEGs.

---

Architecture & Mental Model

The GKE VPC-Native & Datapath V2 Flow

┌────────────────────────────────────────────────────────────────────────────────────────┐
│                                   VPC Subnet (10.100.0.0/20)                           │
│   Primary Range (Node IPs): 10.100.0.0/20                                              │
│   Pod Secondary Range:      10.101.0.0/16                                              │
│   Service Secondary Range:  10.102.0.0/20                                              │
└───────────────────────────────────────────┬────────────────────────────────────────────┘
                                            │
               ┌────────────────────────────┴────────────────────────────┐
               ▼                                                         ▼
    ┌───────────────────────────────┐         ┌───────────────────────────────┐
    │ GKE Node 1 (IP: 10.100.0.5)   │         │ GKE Node 2 (IP: 10.100.0.6)   │
    │ Allocated Pod CIDR:           │         │ Allocated Pod CIDR:           │
    │ 10.101.0.0/24 (256 Pod IPs)   │         │ 10.101.1.0/24 (256 Pod IPs)   │
    │                               │         │                               │
    │  ┌─────────────────────────┐  │         │  ┌─────────────────────────┐  │
    │  │ Pod A (10.101.0.2)      │  │         │  │ Pod B (10.101.1.2)      │  │
    │  └────────────┬────────────┘  │         │  └────────────▲────────────┘  │
    │               │               │         │               │               │
    │               ▼               │         │               │               │
    │  ┌─────────────────────────┐  │         │  ┌────────────┴────────────┐  │
    │  │ Linux Kernel eBPF Hook  │  │         │  │ Linux Kernel eBPF Hook  │  │
    │  │ (Datapath V2 / Cilium)  │──┼─────────┼─►│ (Datapath V2 / Cilium)  │  │
    │  │ BPF Map: Service Routing│  │ Google  │  │ BPF Map: Network Policy│  │
    │  └─────────────────────────┘  │ Network │  └─────────────────────────┘  │
    │  (Zero iptables, Zero NAT)    │ Fabric  │                               │
    └───────────────────────────────┘         └───────────────────────────────┘

• Zero Overlay Encapsulation: Pod A speaks to Pod B using raw IP packets directly across Google’s physical network switches. There is no GRE or VXLAN header overhead, delivering bare-metal network throughput.

---

Core Concepts

1. VPC-Native vs. Routes-Based Networking

Feature	Routes-Based (Legacy)	VPC-Native (Modern Standard)
Mechanisms	Custom static routes in VPC routing table pointing to Node VMs	Subnet Secondary IP Ranges (Alias IPs) allocated directly to Pods
Pod IP Scope	Virtual overlay; invisible to the rest of the VPC	Real, routable RFC 1918 IPs visible to the entire VPC
Max Cluster Size	Limited by VPC route limit (max ~500 nodes)	Scales to 15,000 nodes
Load Balancing	Double-hop: LB ──► Node VM (NodePort) ──► Pod	Single-hop: LB ──► Zonal NEG ──► Direct Pod IP

2. Datapath V2 (Cilium & eBPF Engine)

Datapath V2 is GKE’s modern networking dataplane based on open-source Cilium and the Linux kernel’s extended Berkeley Packet Filter (eBPF):

Why kube-proxy iptables Fails at Scale:

Traditional Kubernetes uses kube-proxy to write iptables rules for every Service. Because iptables evaluates rules sequentially ($O(N)$ algorithmic complexity), a cluster with 5,000 Services creates over 25,000 iptables rules. Every network packet must iterate through thousands of rules, causing severe CPU spikes and packet latency during deployments.

The Datapath V2 eBPF Advantage:

• $O(1)$ Hash Map Lookups: Datapath V2 replaces iptables with BPF hash tables in the Linux kernel. Service lookups complete in constant time ($O(1)$), regardless of whether the cluster has 10 Services or 50,000 Services.
• Native NetworkPolicy Enforcement: Enforces Kubernetes NetworkPolicy rules natively in the kernel without installing Calico or third-party daemonsets.
• Deep Observability: Integrates with Hubble, providing live visibility into DNS lookups, dropped packets, and HTTP layer latency.

3. Container-Native Load Balancing (Zonal NEGs)

In traditional Kubernetes, an external load balancer targets Node VMs on a high NodePort (e.g. 32000). The node receives the packet and uses kube-proxy NAT to forward it across the cluster network to the actual Pod on another node:

• The Double Hop Problem: Incurs extra network latency and obscures client source IP addresses.
• Zonal NEGs (Container-Native): The Google Cloud Load Balancer directly registers the individual Pod IP addresses into a Network Endpoint Group (NEG). Incoming traffic travels directly from Google Front End (GFE) proxies to the Pod container without hitting intermediate nodes!

Traditional Ingress (Double Hop):
Client ──► External LB ──► Node 1 (NodePort) ──► SNAT / Network Hop ──► Node 2 (Pod)

Container-Native Ingress (Single Hop):
Client ──► Google External LB ─────────► Direct to Pod IP on Node 2

---

Gateway API (Next-Gen Kubernetes Ingress)

The Kubernetes Gateway API is the modern successor to traditional Ingress, supported natively on GKE:

GKE GatewayClasses

GatewayClass	Scope	Target Load Balancer
gke-l7-global-external-managed	Global	Global External Application Load Balancer (Anycast IP, Cloud Armor, CDN)
gke-l7-regional-external-managed	Regional	Regional External Application Load Balancer
gke-l7-rilb	Regional	Regional Internal Application Load Balancer (Private VPC traffic)

Production Gateway & HTTPRoute Configuration

# 1. Define the Gateway (Provisions Google Cloud Load Balancer)
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: prod-external-gateway
  namespace: production
spec:
  gatewayClassName: gke-l7-global-external-managed
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      tls:
        mode: Terminate
        certificateRefs:
          - name: prod-api-cert
---
# 2. Define HTTPRoute with Canary Traffic Splitting
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-service-route
  namespace: production
spec:
  parentRefs:
    - name: prod-external-gateway
  hostnames:
    - "api.company.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /v1/payments
      backendRefs:
        # 90% Production stable traffic
        - name: payment-service-v1
          port: 8080
          weight: 90
        # 10% Canary release traffic
        - name: payment-service-v2
          port: 8080
          weight: 10

---

Production gcloud CLI Commands

1. Creating a Production GKE Cluster with Datapath V2 and Custom CIDRs

gcloud container clusters create prod-dataplane-cluster \
  --region=us-central1 \
  --release-channel=regular \
  --network=prod-vpc \
  --subnetwork=prod-us-central1 \
  --cluster-secondary-range-name=gke-pods \
  --services-secondary-range-name=gke-services \
  --enable-ip-alias \
  --enable-dataplane-v2 \
  --dataplane-v2-metrics \
  --dataplane-v2-observability-mode=INTERNAL_VPC_LB \
  --gateway-api=standard \
  --default-max-pods-per-node=32 \
  --enable-private-nodes \
  --master-ipv4-cidr=172.16.0.32/28 \
  --enable-master-authorized-networks \
  --master-authorized-networks=203.0.113.10/32 \
  --num-nodes=2 \
  --enable-autoscaling \
  --min-nodes=2 \
  --max-nodes=10

• --default-max-pods-per-node=32: Allocates a /26 (64 IPs) per node rather than the wasteful default /24 (256 IPs), saving 75% of your secondary Pod CIDR space!

---

Quotas & Limits

Parameter	Limit	Production Guidance
Max pods per cluster	300,000 pods	Bound by secondary CIDR size
Max nodes per cluster	15,000 nodes	Supported in VPC-native mode
Secondary IP ranges per subnet	30 ranges	Plan Pod & Service CIDRs before launch
Master IPv4 CIDR	Exactly /28	Cannot overlap any VPC or VPN CIDR
Default max pods per node	110 (allocates /24)	Reduce to 32 or 64 for CIDR efficiency

---

References

• GKE Networking Overview: https://cloud.google.com/kubernetes-engine/docs/concepts/network-overview
• VPC-Native Clusters Guide: https://cloud.google.com/kubernetes-engine/docs/how-to/alias-ips
• Datapath V2 Documentation: https://cloud.google.com/kubernetes-engine/docs/concepts/about-dataplane-v2
• Gateway API on GKE: https://cloud.google.com/kubernetes-engine/docs/concepts/gateway-api
• Pricing: https://cloud.google.com/kubernetes-engine/pricing

---

Pricing Examples

Scenario 1: Production Multi-Zone GKE Networking Footprint

• 3-Zone GKE cluster running 30 nodes with 600 pods in us-central1.
• In-cluster pod-to-pod networking (same zone): $0.00 (Free).
• Cross-zone internal pod communication: 10 TB / month ($0.01/GB=\ast \ast$100.00 / month**).
• Datapath V2 / eBPF Engine: $0.00 (Included with GKE).
• Container-Native Zonal NEGs: $0.00 (Included with Cloud Load Balancing).
• Total Internal Networking Surcharge: ~$100.00 / month.

Scenario 2: High-Volume Gateway API Ingress

• 1 Global Gateway API Load Balancer handling 250 million HTTP requests / month.
• Ingress and TLS offload: ~$150.00 / month based on processed data volume.
• Elimination of intermediate NodePort hops saves ~15% in intra-cluster cross-zone traffic ($30–$50/mo savings).
• Net Monthly Cost: ~$110.00 / month.

---

Nuggets & Gotchas

1. Pod Secondary Range Cannot Be Resized After Creation: The Pod secondary IP range is hardcoded into the cluster during creation. If your business grows and you run out of Pod IPs in that secondary range, you cannot expand the CIDR. You must deploy a brand-new GKE cluster and migrate workloads. Always size your Pod secondary range conservatively (e.g. /16 or /17).
2. The max-pods-per-node CIDR Multiplier: In GKE, even if a node only runs 5 pods, GKE allocates a full CIDR block twice the size of --max-pods-per-node to every node. If max-pods is 110, it allocates a /24 (256 addresses). If you launch a 100-node cluster, GKE instantly carves out 25,600 Pod IP addresses from your secondary range! Setting --default-max-pods-per-node=32 allocates a /26 (64 IPs), saving 75% of your address space.
3. Datapath V2 Disables Legacy Calico: Enabling Datapath V2 (--enable-dataplane-v2) activates Cilium in the kernel. If you attempt to install Calico or other CNI daemonsets manually via Helm, the cluster will crash with severe BPF map conflicts and broken routing.
4. Zonal NEG Drain Delays on Pod Shutdown: When a Pod is terminated, Google Cloud Load Balancer must drain active connections to the Zonal NEG before the container is killed. If your Pod does not implement a preStop hook with a sleep (e.g. sleep 25), Kubernetes will terminate the container before the Google Front End finishes updating the NEG endpoint list, causing transient HTTP 502 errors for in-flight requests.
5. Private Clusters Block Webhook Admission Controllers by Default: In a private GKE cluster, the master API server runs in a Google-managed VPC. When an admission webhook (e.g. cert-manager, Datadog, Istio) runs on worker nodes on port 8443 or 9443, the API server cannot reach the webhook because the default firewall rule only permits port 443 and 10250. You must add a custom firewall rule allowing traffic from the /28 master CIDR to your webhook ports!

Across the wiki

• Amazon VPC CNI — pod networking (Kubernetes)
• AKS Networking Deep Dive — Azure CNI, CNI Overlay, and Dynamic Pod IP Allocation — pod networking (Azure)
• Network Namespaces — pod networking (Linux)
• Container Networking — pod networking (Containers)