GKE Multi-Cluster Services (MCS) and Multi-Cluster Ingress (MCI) Architecture 🌐🛰️

Modern enterprise Kubernetes architectures often require distributing workloads across multiple GKE clusters to satisfy regulatory data residency mandates, blast-radius isolation, and planetary active-active disaster recovery. Google Kubernetes Engine provides two native primitives for multi-cluster networking: Multi-Cluster Services (MCS), which enables private east-west pod-to-pod communication across clusters using standardized ServiceExport and ServiceImport resources, and Multi-Cluster Ingress (MCI), which deploys a single Google Cloud Global External Application Load Balancer that intelligently routes traffic to the nearest healthy cluster.

---

1. Multi-Cluster Architecture & Global Traffic Flow

Multi-cluster networking on GKE leverages Google Cloud Fleets (formerly Anthos Environments) to unify cluster identities under a shared trust domain.

                           GLOBAL EXTERNAL TRAFFIC (ANYCAST VIP)
                                             │
                                             ▼
       ┌────────────────────────────────────────────────────────────────────────┐
       │                 MULTI-CLUSTER INGRESS (MCI) CONTROLLER                 │
       │                   (Global Application Load Balancer)                   │
       │  - Single Anycast IP advertised across 100+ Google Edge PoPs           │
       │  - Routes to nearest region with available Pod capacity                │
       │  - Zero-downtime automated failover during regional datacenter outages │
       └───────────────────┬────────────────────────────────┬───────────────────┘
                           │                                │
        US-EAST-4 REGION   │ Lowest Latency                 │ EUROPE-WEST1 REGION
                           ▼                                ▼
       ┌───────────────────────────────┐        ┌───────────────────────────────┐
       │ GKE CLUSTER 1: PROD-US        │        │ GKE CLUSTER 2: PROD-EU        │
       │                               │        │                               │
       │  ┌─────────────────────────┐  │        │  ┌─────────────────────────┐  │
       │  │ User Pods (Zonal NEGs)  │  │        │  │ User Pods (Zonal NEGs)  │  │
       │  └────────────┬────────────┘  │        │  └────────────┬────────────┘  │
       │               │               │        │               │               │
       │  ┌────────────▼────────────┐  │        │  ┌────────────▼────────────┐  │
       │  │ ServiceExport           │  │        │  │ ServiceExport           │  │
       │  │ `payment-service`       │  │        │  │ `payment-service`       │  │
       │  └────────────┬────────────┘  │        │  └────────────┬────────────┘  │
       └───────────────┼───────────────┘        └───────────────┼───────────────┘
                       │                                        │
       ════════════════╪════════════════════════════════════════╪════════════════
       SHARED VPC (Private Google Cloud Fiber Backplane - No NAT)
                       │                                        │
                       ▼                                        ▼
       ┌────────────────────────────────────────────────────────────────────────┐
       │                MULTI-CLUSTER SERVICES (MCS) DNS RESOLUTION             │
       │                                                                        │
       │   Query: `payment-service.finance.svc.clusterset.local`                │
       │   - In-cluster CoreDNS resolves to healthy Pod endpoints in BOTH       │
       │     Cluster 1 and Cluster 2 simultaneously.                            │
       └────────────────────────────────────────────────────────────────────────┘

Core Architecture Constructs

1. GKE Fleet: The central organizational boundary that binds multiple GKE clusters together. Clusters in the same Fleet share a common Sameness of Namespaces (e.g., namespace finance in Cluster 1 is identical in trust to namespace finance in Cluster 2).
2. Multi-Cluster Services (MCS - KEP-1645):
   • ServiceExport: Applied to a local Service. Instructs the MCS controller to export the service across all member clusters in the Fleet.
   • ServiceImport: Automatically generated by the controller in all peer clusters. Represents the virtual multi-cluster IP.
   • clusterset.local DNS Zone: Workloads query my-svc.my-ns.svc.clusterset.local. CoreDNS dynamically discovers and load balances requests to pod endpoints in any healthy cluster across the fleet.
3. Multi-Cluster Ingress (MCI): Deploys a Google Cloud Global External Application Load Balancer with Zonal NEGs in all participating clusters. The config cluster acts as the centralized control plane where MultiClusterIngress and MultiClusterService manifests are applied.

---

2. Global Load Balancing & Proximity Routing Mechanics

MCI routes incoming requests based on geographical proximity and backend capacity:

1. Edge Anycast Termination: A client in London terminates TLS at Google’s London edge PoP.
2. Proximity Forwarding: The load balancer prefers routing to the local europe-west1 cluster.
3. Spillover & Capacity Awareness: If the europe-west1 cluster reaches maximum configured capacity (max queries per second or max connections) or becomes unhealthy, traffic seamlessly spills over to the us-east4 cluster over Google’s internal private fiber backbone without dropping TCP connections.

---

3. Production Deployment & CLI Operations (gcloud & kubectl)

1. Register GKE Clusters into a Unified Fleet

# Enable Multi-Cluster APIs
gcloud services enable \
    gkehub.googleapis.com \
    multiclusteringress.googleapis.com \
    multiclusterservicediscovery.googleapis.com \
    --project=core-infrastructure-prod
 
# Register Cluster 1 (US) to the Fleet
gcloud container fleet memberships register gke-prod-us \
    --gke-cluster=us-central1/prod-cluster-us \
    --enable-workload-identity \
    --project=core-infrastructure-prod
 
# Register Cluster 2 (EU) to the Fleet
gcloud container fleet memberships register gke-prod-eu \
    --gke-cluster=europe-west1/prod-cluster-eu \
    --enable-workload-identity \
    --project=core-infrastructure-prod

2. Enable Multi-Cluster Services (MCS) Discovery

# Enable MCS feature on the Fleet
gcloud container fleet multi-cluster-services enable \
    --project=core-infrastructure-prod
 
# Grant required IAM permissions to MCS service accounts
gcloud projects add-iam-policy-binding core-infrastructure-prod \
    --member="serviceAccount:core-infrastructure-prod.svc.id.goog[gke-mcs/gke-mcs-importer]" \
    --role="roles/compute.networkViewer"

3. Export a Service for Cross-Cluster Consumption

On Cluster 1 (prod-cluster-us), deploy the service and export it:

apiVersion: v1
kind: Service
metadata:
  name: payment-service
  namespace: finance
spec:
  selector:
    app: payment-processor
  ports:
    - port: 8080
      targetPort: 8080
---
apiVersion: net.gke.io/v1
kind: ServiceExport
metadata:
  name: payment-service
  namespace: finance

Apply to Cluster 1:

kubectl apply -f payment-service-export.yaml --context=gke-prod-us

4. Consume the Cross-Cluster Service from Cluster 2

Inside Cluster 2 (prod-cluster-eu), pods can now make calls directly using the clusterset.local domain:

# Verify ServiceImport was generated in Cluster 2
kubectl get serviceimport payment-service -n finance --context=gke-prod-eu
 
# Pod in Cluster 2 makes a private gRPC / HTTP call to Cluster 1's pods:
curl http://payment-service.finance.svc.clusterset.local:8080/healthz

5. Deploy Multi-Cluster Ingress (MCI) for Global Failover

Designate Cluster 1 as the Config Cluster:

gcloud container fleet ingress enable \
    --config-membership=gke-prod-us \
    --project=core-infrastructure-prod

Deploy mci-deployment.yaml to the config cluster:

apiVersion: networking.gke.io/v1
kind: MultiClusterIngress
metadata:
  name: global-storefront-mci
  namespace: storefront
spec:
  template:
    spec:
      backend:
        serviceName: storefront-mcs
        servicePort: 80
---
apiVersion: networking.gke.io/v1
kind: MultiClusterService
metadata:
  name: storefront-mcs
  namespace: storefront
spec:
  template:
    spec:
      selector:
        app: storefront-ui
      ports:
        - name: http
          port: 80
          targetPort: 8080

Apply to Config Cluster:

kubectl apply -f mci-deployment.yaml --context=gke-prod-us

---

4. Quotas, Performance, and Configuration Limits

Parameter / Dimension	Default Limit	Engineering Guidance
Max Clusters per Fleet	Up to 100 clusters	Multi-region, multi-cloud unified fleet
Cross-Cluster Latency	Governed by GCP backbone	Sub-millisecond jitter; zero public internet
ServiceExport Propagation	~10 to 30 seconds	Time for DNS zone update in peer clusters
MCI VIP Allocation	Global IPv4 Anycast	Routes to nearest region with available pods
VPC Requirement	Shared VPC or Flat Peering	Requires non-overlapping Pod and Service CIDRs

---

5. Official References & Documentation

• GKE Multi-Cluster Services (MCS) Documentation
• GKE Multi-Cluster Ingress (MCI) Architecture
• Google Cloud Fleets Architectural Overview
• Kubernetes Multi-Cluster Services API (KEP-1645)
• GKE Enterprise / Fleet Pricing

---

6. Realistic Pricing Scenarios

Pricing components:

1. Multi-Cluster Ingress (MCI): Included in GKE Enterprise (Anthos) tier, or billed as a standalone feature ($0.50percluster-houracrossallregisteredclusters=$365/month per cluster).
2. Google Cloud Global Application Load Balancer: Standard forwarding rules ($18.25/mo)+$0.008/GB processed.
3. Cross-Region Network Traffic: Standard GCP inter-regional data transfer ($0.02-$0.05/GB) when requests cross regional boundaries.

Scenario A: Active-Active Multi-Region Public API (2 Clusters)

• Architecture:
  • 2 Regional GKE Clusters (us-central1 and europe-west1).
  • MCI enabled with Global Anycast Load Balancer.
  • Data transfer: 20,000 GB (20 TB) monthly ingress/egress.
  • 95% of traffic served locally within its origin continent; 5% inter-continental failover spillover (1,000 GB).
• Monthly Cost Calculation:
  • Standalone MCI Fee (2 clusters): 2 × $0.50/hr\times 730hrs=\ast \ast$730.00**
  • Load Balancer Forwarding Rule: $0.025/hr\times 730hrs=\ast \ast$18.25**
  • Data Processed Fee: 20,000 GB × $0.008/GB=\ast \ast$160.00**
  • Inter-Region Egress Spillover (1,000 GB): 1,000 GB × $0.02/GB=\ast \ast$20.00**
• Total Monthly Cost: $928.25 / month

Scenario B: Private East-West Multi-Cluster Microservices (MCS Only)

• Architecture:
  • 3 GKE Clusters connected to a single Shared VPC.
  • Internal cross-cluster microservice queries via clusterset.local.
  • Zero external load balancers.
  • Inter-cluster data transfer: 5,000 GB cross-region.
• Monthly Cost Calculation:
  • Multi-Cluster Services (MCS) Platform Fee: $0.00 (Included free with GKE).
  • Inter-Region Network Egress: 5,000 GB × $0.02/GB=\ast \ast$100.00**
• Total Monthly Cost: $100.00 / month

---

7. Battle-Tested Nuggets & Production Gotchas

1. The Overlapping Pod CIDR Disaster: Multi-Cluster Services (MCS) routes traffic directly from a pod in Cluster A to a pod in Cluster B over the Google Cloud VPC network without Network Address Translation (NAT). If both clusters were provisioned with the default Pod IP range 10.4.0.0/14, pods will have identical IP addresses across clusters. When a pod in Cluster A attempts to route to Cluster B, the Linux kernel detects the destination IP as local and drops the packet. Every cluster participating in an MCS Fleet must have completely distinct, non-overlapping Pod and Service secondary subnet CIDRs.
2. The Single Config Cluster Single-Point-of-Failure Myth: When deploying Multi-Cluster Ingress (MCI), one cluster is designated as the config-membership cluster. If that config cluster goes down, the existing Global Load Balancer continues to route traffic to healthy pods in all other clusters without interruption. You only lose the ability to deploy new MCI manifests or modify routing rules until the config cluster recovers.
3. CoreDNS Stub Domain Caching Gotcha: MCS creates a custom DNS domain clusterset.local. If you have heavily customized in-cluster CoreDNS configurations or upstream DNS forwarders that do not forward .clusterset.local to the GKE MCS DNS provider, workloads will receive NXDOMAIN when attempting to query cross-cluster services.
4. Namespace Sameness Enforcement: In GKE Fleets, if a service is exported in namespace analytics, the receiving pods in another cluster must also reside in namespace analytics. If team A deploys in analytics-prod and exports a service, team B cannot import it into namespace reporting. Design your enterprise namespace naming conventions uniformly across all clusters from day one.
5. Session Affinity Does Not Span Clusters: While GKE load balancers support cookie-based or client-IP-based session affinity, session affinity is pinned to a specific backend service within a specific cluster. If an application requires sticky server sessions and the primary regional cluster experiences performance degradation, MCI will reroute traffic to the secondary cluster, breaking the user’s session and forcing re-authentication. State must be externalized (e.g., in Memorystore Redis) for multi-cluster architectures.
6. Firewall Rules for Cross-Cluster Health Checking: When using Multi-Cluster Ingress, Google Cloud health check probers (130.211.0.0/22 and 35.191.0.0/16) must be able to reach the node ports and pod ports of all member clusters. If a cluster’s VPC firewall rule only whitelists traffic from its own subnet, the global load balancer will declare all backends in that cluster UNHEALTHY and refuse to route traffic to it.

Across the wiki

• Azure Kubernetes Fleet Manager — Multi-Cluster Governance, Staged Upgrades, and Multi-Cluster Services (MCS) — multi-cluster (Azure)
• Multi-Cluster — multi-cluster (Kubernetes)
• Advanced EKS Networking — multi-cluster (Kubernetes)