cloudnative wiki

GKE Cost Optimization, FinOps, and GKE Cost Allocation Architecture

8 min read

GKE Cost Optimization, FinOps, and GKE Cost Allocation Architecture 💰📊

Kubernetes excels at abstracting hardware infrastructure, but this abstraction frequently obscures financial visibility. In a large shared GKE cluster, platform engineering teams often cannot determine which team, microservice, or environment generated a $50,000 cloud bill. GKE Cost Allocation solves this by breaking down physical Compute Engine and Persistent Disk infrastructure costs by Kubernetes Namespace and Pod Labels, streaming granular cost telemetry into BigQuery. Combining Cost Allocation with Spot VM pools, GKE-specific Committed Use Discounts (CUDs), and Slack Capacity Elimination delivers world-class Kubernetes FinOps governance.

1. Architecture: GKE Cost Allocation & BigQuery Telemetry Pipeline

GKE Cost Allocation instruments the GKE control plane to track resource consumption per container, joining Kubernetes object metadata with Google Cloud billing SKUs in real time.

                           KUBERNETES WORKLOAD FLEET
       ┌────────────────────────┐         ┌────────────────────────┐
       │ Namespace: `checkout`  │         │ Namespace: `analytics` │
       │ Labels: team=cart      │         │ Labels: team=data      │
       │ Request: 8 CPU, 32G RAM│         │ Request: 32 CPU, 128G  │
       └───────────┬────────────┘         └───────────┬────────────┘
                   │                                  │
                   └─────────────────┬────────────────┘
                                     │ Actual Resource Usage vs Requests
                                     ▼
       ┌────────────────────────────────────────────────────────────────────────┐
       │                   GKE COST ATTRIBUTION METERING AGENT                  │
       │  - Measures requested vs unallocated (slack) CPU, Memory, and Disk     │
       │  - Attaches Kubernetes metadata: Cluster, Namespace, Pod, Labels       │
       └───────────────────────────────────┬────────────────────────────────────┘
                                           │ Hourly Streaming Export
       ════════════════════════════════════╪═════════════════════════════════════
       GOOGLE CLOUD BILLING INFRASTRUCTURE │
                                           ▼
       ┌────────────────────────────────────────────────────────────────────────┐
       │                 BIGQUERY DETAILED CLOUD BILLING EXPORT                 │
       │                                                                        │
       │  ┌──────────────────────────────────────────────────────────────────┐  │
       │  │ `gcp_billing_export_resource_v1_*`                               │  │
       │  │ - Line-item unblended costs                                      │  │
       │  │ - Amortized CUD & SUD discounts                                  │  │
       │  │ - Exact dollar cost per Kubernetes namespace & microservice      │  │
       │  └──────────────────────────────────┬───────────────────────────────┘  │
       └─────────────────────────────────────┼──────────────────────────────────┘
                                             │ SQL Queries / Looker Studio
                                             ▼
       ┌────────────────────────────────────────────────────────────────────────┐
       │                     FINOPS CHARGEBACK & SHOWBACK                       │
       │      "Team Cart: $1,240/mo | Team Data: $8,400/mo | Idle Slack: $410"  │
       └────────────────────────────────────────────────────────────────────────┘

The Three Components of GKE Cost

When analyzing a GKE cluster invoice, cost divides into three distinct categories:

  1. Utilized Workload Cost: Compute and memory actively consumed by running containers.
  2. Requested Slack Capacity: Compute and memory requested by pods but sitting idle because developers over-provisioned requests out of caution.
  3. Unallocated Cluster Slack: Physical VM cores and RAM in the node pool that cannot be scheduled because remaining capacity is fragmented across nodes.

2. Granular SQL FinOps Queries in BigQuery

1. Calculate Exact Monthly Spend by Kubernetes Namespace

SELECT
  labels.value AS k8s_namespace,
  ROUND(SUM(cost), 2) AS raw_cost,
  ROUND(SUM(cost + IFNULL((SELECT SUM(c.amount) FROM UNNEST(credits) c), 0)), 2) AS net_cost
FROM
  `billing_export.gcp_billing_export_resource_v1_XXXXXX_XXXXXX_XXXXXX`,
  UNNEST(labels) as labels
WHERE
  labels.key = "k8s-namespace"
  AND _PARTITIONDATE >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
GROUP BY
  1
ORDER BY
  net_cost DESC;

2. Identify Top 5 Overprovisioned (Slack) Microservices

SELECT
  labels.value AS pod_name,
  ROUND(SUM(cost), 2) AS total_pod_cost
FROM
  `billing_export.gcp_billing_export_resource_v1_XXXXXX_XXXXXX_XXXXXX`,
  UNNEST(labels) as labels
WHERE
  labels.key = "k8s-pod"
  AND _PARTITIONDATE >= DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY)
GROUP BY
  1
ORDER BY
  total_pod_cost DESC
LIMIT 5;

3. Production Deployment & CLI Operations (gcloud)

1. Enable GKE Cost Allocation on Production Cluster

gcloud container clusters update prod-regional-cluster \
    --region=us-central1 \
    --enable-cost-allocation \
    --project=core-infrastructure-prod

(GKE begins tagging GCE billing exports with Kubernetes namespaces and pod labels within 24 hours).

2. Purchase Flexible Spend-Based Committed Use Discount (CUD)

Flexible Spend CUDs provide up to 46% savings across GKE Autopilot and GKE Standard worker nodes across any machine family globally:

# Purchase a 1-year flexible commitment of $100/hr spend
gcloud compute commitments create gke-flex-cud-1yr \
    --region=us-central1 \
    --plan=TWELVE_MONTH \
    --category=COMPUTE \
    --type=FLEXIBLE \
    --amount=100.00 \
    --project=billing-admin-prod

3. Deploy Spot Node Pool with Automated Scale-to-Zero

gcloud container node-pools create spot-stateless-pool \
    --cluster=prod-regional-cluster \
    --region=us-central1 \
    --machine-type=e2-standard-8 \
    --spot \
    --num-nodes=0 \
    --enable-autoscaling \
    --min-nodes=0 \
    --max-nodes=20 \
    --node-labels="cloud.google.com/gke-spot=true,workload=stateless" \
    --project=core-infrastructure-prod

4. Deploy Overprovisioning (Pause) Pods to Prevent Cold Starts

Create overprovisioning-pause.yaml:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: overprovisioning-priority
value: -1 # Lowest priority: gets evicted first
globalDefault: false
description: "Used by ballooning pause pods to hold spare node capacity"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-overprovisioning-balloon
  namespace: kube-system
spec:
  replicas: 4 # Holds 4 nodes worth of buffer capacity
  selector:
    matchLabels:
      app: overprovisioning-balloon
  template:
    metadata:
      labels:
        app: overprovisioning-balloon
    spec:
      priorityClassName: overprovisioning-priority
      terminationGracePeriodSeconds: 0
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.9
          resources:
            requests:
              cpu: "3" # Reserves 3 cores per node
              memory: "12Gi"

Apply overprovisioning:

kubectl apply -f overprovisioning-pause.yaml

4. Quotas, Performance, and Configuration Limits

Parameter / DimensionLimit / MetricFinOps Recommendation
Billing Export Latency2 to 6 hoursBigQuery billing data is near-real-time, not instant
Max Pod Labels Exported64 labels per resourceStandardize labels (app, env, team, cost-center)
Flexible CUD DiscountUp to 46% (3-year term)Commit to 70% of historical baseline compute trough
Spot VM Discount60% to 90% discountIdeal for stateless web replicas and batch queues
Unallocated Slack Target< 15% of cluster spendUse optimize-utilization profile or GKE Autopilot

5. Official References & Documentation

6. Realistic Pricing Scenarios

FinOps optimization systematically cuts cluster waste:

  1. Standard On-Demand Compute: Baseline retail pricing.
  2. CUDs: Commitments delivering 37% (1-yr) to 57% (3-yr) discounts.
  3. Spot VMs: Up to 90% discount on interruptible capacity.

Scenario A: Un-Optimized Enterprise GKE Estate (Before FinOps)

  • Cluster Profile:
    • 100 n2-standard-8 nodes ($0.388/hr each) running on 100% on-demand pricing.
    • Average pod CPU utilization: 18% (Massive overprovisioned slack capacity).
    • Monthly Compute Bill: 100 nodes × 28,324.00 / month**.
    • Cluster Management Fee: $73.00.
    • Total Monthly Spend: **340,764/year).

Scenario B: Fully Optimized FinOps GKE Estate (After Optimization)

  • Strategic Actions Applied:
    1. Rightsizing via VPA: Downsized pod requests by 40%, reducing required nodes from 100 to 60.
    2. Committed Use Discounts: Purchased a 3-year Flexible CUD for 40 nodes baseline (46% discount = $0.210/hr).
    3. Spot Node Pool: Moved 20 stateless/batch nodes to Spot VMs (70% discount = $0.116/hr).
  • Monthly Cost Calculation:
    • CUD Nodes (40 nodes): 40 × 6,132.00**
    • Spot Nodes (20 nodes): 20 × 1,693.60**
    • Cluster Management Fee: $73.00
  • Optimized Monthly Spend: $7,898.60 / month
  • Net Annual Savings: (\28,397.00 - $7,898.60) \times 12 = \mathbf{$245,980.80 / year saved}$ (~72% cost reduction).

7. Battle-Tested Nuggets & Production Gotchas

  1. Labels Added After Pod Deployment Do Not Backfill Billing: GKE Cost Allocation attaches Kubernetes labels to billing records at the moment the usage occurs. If a pod runs for 3 weeks without a cost-center label, and you patch the Deployment to add the label today, BigQuery cannot retroactively attribute the past 3 weeks of spend. Enforce mandatory pod labels at admission time using OPA Gatekeeper or Kyverno.
  2. The “Requests Equal Limits” Cost Inflation Trap: In GKE Standard, setting pod requests = limits gives pods the Guaranteed QoS class, preventing throttling. However, if a developer sets both to 8 vCPUs for an app that averages 200m CPU, Kubernetes locks up 8 physical cores on that node. The node pool autoscales to accommodate the inflated requests, running dozens of empty VMs. FinOps best practice: set requests based on P95 historical usage, leaving limits higher to handle burst spikes.
  3. Commitment Break-Even Calculation (The 70% Baseline Rule): A 1-year CUD provides ~37% savings; a 3-year CUD provides ~57% savings. However, CUDs bill 24 hours a day, 365 days a year, whether instances are running or not. If your workload shuts down on weekends (running only 45% of the month), on-demand or Spot VMs are actually cheaper than a 1-year CUD. Only purchase CUDs for workloads running of the billing month.
  4. Spot Node Evictions Triggering On-Demand Failover Churn: If you configure a cluster to fall back to on-demand nodes when Spot capacity is preempted, ensure your Cluster Autoscaler includes scaling priority policies (expander: priority). Without priority expanders, CA may continue spinning up expensive on-demand nodes and refuse to return to Spot nodes when Spot capacity recovers, leaving workloads on high-cost compute indefinitely.
  5. Orphaned Persistent Disks in Scale-Down Events: When Cluster Autoscaler terminates a node, or when a developer deletes a namespace containing StatefulSets, the underlying GCE Persistent Disks are retained by default. If a cluster churns through 50 stateful test jobs a week, hundreds of detached 100 GB SSD disks accumulate in the GCP project, costing $17/disk-month in silent waste. Run automated cron scripts to delete detached disks older than 7 days (gcloud compute disks list --filter="users:-").
  6. Autopilot vs Standard Cost Inflection Point: GKE Autopilot charges for exact Pod requests without node overhead, making it dramatically cheaper for small clusters (< 20 nodes) or variable workloads where node slack would otherwise be wasted. However, for a stable, highly optimized enterprise cluster running at CPU/RAM bin-packing density with 3-year CUDs, GKE Standard can be 15% to 25% cheaper than Autopilot because you leverage hardware economies of scale. Perform a FinOps audit annually to determine the optimal compute model.

Across the wiki

Graph View

Backlinks