GKE Fleets, Cloud Service Mesh (formerly ASM), and Policy Controller Governance 🌐🏛️

As Kubernetes adoption scales within the enterprise, organizations inevitably transition from managing isolated individual clusters to operating an entire Fleet of clusters spanning multiple GCP projects, regions, on-premises datacenters, and edge environments. Google Cloud Fleets unifies multiple Kubernetes clusters into a single logical management plane. It establishes Namespace Sameness, automates GitOps configuration deployment via Config Sync, enforces enterprise security baselines via Policy Controller (OPA Gatekeeper), and provides managed service mesh capabilities via Cloud Service Mesh (formerly Anthos Service Mesh / ASM and Traffic Director).

NOTE

Product Architecture & Lineage: Google Cloud has streamlined its multi-cluster and service mesh portfolio. The legacy “Anthos” umbrella was rebranded to GKE Enterprise. Today, foundational Fleet management, Config Sync GitOps, and Policy Controller are native capabilities of GKE, while Cloud Service Mesh serves as the unified Istio-powered control plane available standalone or within enterprise licensing.

---

1. Architecture: The GKE Fleet Control Plane

A GKE Fleet abstracts physical cluster boundaries, providing centralized governance, identity federation, and global traffic management.

                           ENTERPRISE FLEET CONTROL PLANE
                                          │
            ┌─────────────────────────────┼─────────────────────────────┐
            │                             │                             │
    CONFIG SYNC (GitOps)          POLICY CONTROLLER (OPA)       ANTHOS SERVICE MESH (ASM)
    - Git Repo: Single Source     - Enforces CIS Benchmarks     - Managed Istio Control Plane
    - Auto-reconciles manifests   - Blocks privileged pods      - Zero-trust mTLS encryption
    - Multi-repo tenant configs   - Validates resource quotas   - Distributed tracing & SLOs
            │                             │                             │
            └─────────────────────────────┼─────────────────────────────┘
                                          │ Continuous Reconciliation
                                          ▼
       ┌────────────────────────────────────────────────────────────────────────┐
       │                   GKE FLEET MEMBERSHIP FEDERATION                      │
       │                   (Unified Workload Identity Pool)                     │
       └───────────────────┬────────────────────────────────┬───────────────────┘
                           │                                │
        US REGIONAL CLUSTER│                                │ EUROPE REGIONAL CLUSTER
                           ▼                                ▼
       ┌───────────────────────────────┐        ┌───────────────────────────────┐
       │ GKE MEMBER CLUSTER: PROD-US   │        │ GKE MEMBER CLUSTER: PROD-EU   │
       │                               │        │                               │
       │  ┌─────────────────────────┐  │        │  ┌─────────────────────────┐  │
       │  │ Namespace: `finance`    │  │        │  │ Namespace: `finance`    │  │
       │  │ (Identical Trust Domain)│  │        │  │ (Identical Trust Domain)│  │
       │  └────────────┬────────────┘  │        │  └────────────┬────────────┘  │
       │               │ Envoy Proxy   │        │               │ Envoy Proxy   │
       │  ┌────────────▼────────────┐  │        │  ┌────────────▼────────────┐  │
       │  │ Istio Sidecar (mTLS)    │◄─┼────────┼─►│ Istio Sidecar (mTLS)    │  │
       │  └─────────────────────────┘  │        │  └─────────────────────────┘  │
       └───────────────────────────────┘        └───────────────────────────────┘

Core Architecture Constructs

1. Fleet Membership: Clusters (GKE, AWS EKS, Azure AKS, or bare metal) are registered as members of a Fleet. Membership establishes a shared Workload Identity Pool (<project-id>.hub.id.goog), enabling pods to authenticate to peer clusters without sharing long-lived secrets.
2. Namespace Sameness: Within a Fleet, a namespace with the same name across different clusters is treated as belonging to the same tenant and sharing the same permissions and policies.
3. Anthos Service Mesh (ASM): A Google-managed distribution of open-source Istio. ASM provisions and upgrades the Istiod control plane, manages mTLS certificate rotation via Google Cloud Certificate Authority Service (CAS), and collects Layer 7 telemetry without customer intervention.
4. Policy Controller: Fully managed Kubernetes Open Policy Agent (OPA) Gatekeeper service. It intercepts Kubernetes API requests via admission webhooks and audits running pods against hundreds of built-in compliance templates (PCI-DSS, NIST 800-53, CIS Kubernetes Benchmark).

---

2. Policy Controller & OPA Gatekeeper Guardrails

Policy Controller enforces declarative constraints using the Rego language:

• Enforcement Actions:
  • deny: Blocks any kubectl apply or CI/CD deployment violating the rule.
  • warn: Allows deployment but displays a warning message in the terminal.
  • dryrun: Audits the cluster and logs violations without disrupting workloads.
• Built-in Template Library: Contains pre-configured constraints for disallowing privileged containers, enforcing read-only root filesystems, requiring specific pod labels, and blocking public LoadBalancer services.

---

3. Production Deployment & CLI Operations (gcloud & kubectl)

1. Register Clusters into a Fleet

# Enable Fleet and Anthos APIs
gcloud services enable \
    gkehub.googleapis.com \
    anthos.googleapis.com \
    mesh.googleapis.com \
    anthospolicycontroller.googleapis.com \
    --project=core-infrastructure-prod
 
# Register GKE cluster as a Fleet member
gcloud container fleet memberships register gke-prod-us \
    --gke-cluster=us-central1/prod-regional-cluster \
    --enable-workload-identity \
    --project=core-infrastructure-prod

2. Enable Managed Anthos Service Mesh (ASM) with Google CA

# Enable Anthos Service Mesh feature on the Fleet
gcloud container fleet mesh enable --project=core-infrastructure-prod
 
# Update cluster membership to enable Managed ASM with in-cluster Envoy injection
gcloud container fleet mesh update \
    --management=AUTOMATIC \
    --memberships=gke-prod-us \
    --project=core-infrastructure-prod

Enable sidecar injection on an application namespace:

kubectl label namespace e-commerce istio.io/rev=asm-managed --overwrite

3. Deploy Zero-Trust mTLS PeerAuthentication via ASM

Enforce STRICT mutual TLS (mTLS) across all pods in the e-commerce namespace:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default-strict-mtls
  namespace: e-commerce
spec:
  mtls:
    mode: STRICT # Rejects all non-mTLS plaintext traffic

Apply PeerAuthentication:

kubectl apply -f default-strict-mtls.yaml

4. Enable Policy Controller (OPA Gatekeeper) with Built-in CIS Templates

# Enable Policy Controller on the Fleet
gcloud container fleet policycontroller enable --project=core-infrastructure-prod
 
# Apply default security constraint bundles (CIS Kubernetes Benchmark)
gcloud container fleet policycontroller update \
    --memberships=gke-prod-us \
    --bundle=cis-k8s-v1.5.1 \
    --project=core-infrastructure-prod

5. Deploy Custom Policy Controller Constraint (Require Pod Labels)

Create require-team-label-constraint.yaml:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: require-team-and-env-labels
spec:
  enforcementAction: deny # Hard block on violations
  match:
    kinds:
      - apiGroups: ["apps"]
        kinds: ["Deployment", "StatefulSet"]
    namespaces:
      - "e-commerce"
      - "finance"
  parameters:
    labels:
      - key: "team"
      - key: "env"

Apply Constraint:

kubectl apply -f require-team-label-constraint.yaml
 
# Test constraint: submitting a deployment without 'team' label fails with:
# Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request

---

4. Quotas, Performance, and Configuration Limits

Parameter / Dimension	Standard Limit / Quota	Engineering Guidance
Fleet Member Clusters	Up to 100 clusters	Multi-cloud and multi-region unified fleet
ASM Control Plane Overhead	Managed by Google	Google SREs manage the istiod control plane
Envoy Sidecar Footprint	~50 MiB RAM, 100m CPU	Inject sidecars only where L7 security is required
Policy Controller Rules	100+ active constraints	Sub-millisecond admission evaluation latency
GitOps Reconcile Cadence	Continuous (~15 seconds)	Managed by Config Sync rootsync / reposync

---

5. Official References & Documentation

• GKE Fleets (Anthos) Architectural Overview
• Anthos Service Mesh (ASM) Managed Overview
• Policy Controller Documentation
• Config Sync GitOps Architecture
• GKE Enterprise Edition Pricing

---

6. Realistic Pricing Scenarios

Pricing structure:

1. GKE Enterprise Edition (Anthos): Billed as a unified subscription of **$0.00822pervCPU-hour\ast \ast acrossallmanagedworkernodesinthefleet($6.00 per vCPU-month).
2. Includes: Multi-Cluster Ingress, Anthos Service Mesh, Policy Controller, Config Sync, and Cloud Service Mesh telemetry dashboards.
3. Standalone Pricing: ASM and Policy Controller can be enabled standalone with pay-per-use billing.

Scenario A: Enterprise Microservice Mesh (10 Clusters, 800 vCPUs)

• Fleet Profile:
  • 10 Regional GKE Clusters running 800 total vCPUs.
  • Enterprise requirements: STRICT mTLS encryption across all microservices, automated Config Sync GitOps, and CIS compliance enforcement via Policy Controller.
  • Organization subscribes to GKE Enterprise Edition.
• Monthly Cost Calculation:
  • GKE Enterprise Surcharge: 800 vCPUs × $0.00822/vCPU-hr\times 730hrs=\ast \ast$4,800.48**
  • Standard Cluster Management Fees (10 clusters): 10 × $73.00=\ast \ast$730.00**
  • Compute Engine Infrastructure (VMs and Disks): Standard GCP VM rates.
• Total Management & Mesh Cost: $5,530.48 / month

Scenario B: Standalone Policy Controller Governance (Zero-Mesh Cluster)

• Cluster Profile:
  • 1 GKE cluster with 20 nodes (80 vCPUs).
  • Organization only requires OPA Gatekeeper security guardrails without Service Mesh.
  • Policy Controller enabled standalone without full GKE Enterprise package.
• Monthly Cost Calculation:
  • Policy Controller Standalone: Included with GKE or low-tier per-vCPU fee (~$0.003/vCPU-hr).
  • Cost: 80 vCPUs × $0.003/hr\times 730hrs=\ast \ast$175.20 / month**.
• Total Monthly Cost: $175.20 / month

---

7. Battle-Tested Nuggets & Production Gotchas

1. The Managed ASM Automatic Upgrade Revision Drift: When Managed Anthos Service Mesh is configured with --management=AUTOMATIC, Google Cloud automatically upgrades the Istio control plane. However, existing application pods retain their existing Envoy sidecar containers until the pod is restarted. If microservices are not restarted for 6 months, their Envoy sidecars will drift across multiple minor versions, eventually breaking communication with the newer control plane. Enforce periodic rolling restarts of application workloads following mesh upgrades.
2. STRICT mTLS Rejection on Legacy External Callers: When configuring PeerAuthentication with mode: STRICT, all unencrypted plaintext TCP connections are dropped immediately at the Envoy sidecar. If an external service (like a legacy VM in the VPC or a non-meshed third-party tool) attempts to query a pod directly, the connection terminates with connection reset by peer or SSL alert number 42. Use mode: PERMISSIVE during initial migration to allow both mTLS and plaintext traffic while auditing telemetry.
3. Policy Controller Admission Webhook Timeout Locks Cluster: If the Policy Controller validating admission webhook experiences high latency or if all Policy Controller pods crash, the Kubernetes API server behavior depends on failurePolicy. If set to Fail (Fail-Closed), all future kubectl apply, Helm deployments, and pod creation requests across the entire cluster will fail with timeout errors. For non-critical policies, set failurePolicy: Ignore (Fail-Open) in non-production environments.
4. Fleet Registration Requires Unique Membership Names: When registering clusters to a Fleet, every membership name must be globally unique within the Google Cloud project. Attempting to register two clusters with the name production-cluster (even if located in different regions like us-central1 and europe-west1) will result in an ALREADY_EXISTS error. Use regional suffixes (gke-prod-us-central1 and gke-prod-europe-west1).
5. Config Sync Git Repository Access Token Expiration: When configuring Config Sync to pull Kubernetes manifests from a private GitHub or GitLab repository, engineers often use Personal Access Tokens (PATs). When the PAT expires after 90 days, Config Sync halts synchronization silently, leaving cluster configurations out of sync with Git without alerting developers. Always authenticate Config Sync via Workload Identity OIDC federation or deploy an automated secret refresher.
6. Envoy Sidecar Memory Leaks on High-Connection Gateways: When microservices handle tens of thousands of concurrent WebSocket connections or long-lived gRPC streams, the Envoy sidecar container (istio-proxy) can consume significant RAM tracking connection state. If the pod specification does not declare explicit resource limits for the injected sidecar (sidecar.istio.io/proxyMemoryLimit), Envoy can consume all node memory, triggering the Linux kernel Out-Of-Memory (OOMKilled) killer on your application container.

Across the wiki

• Azure Kubernetes Fleet Manager — Multi-Cluster Governance, Staged Upgrades, and Multi-Cluster Services (MCS) — multi-cluster (Azure)
• Multi-Cluster — multi-cluster (Kubernetes)
• Advanced EKS Networking — multi-cluster (Kubernetes)