cloudnative wiki

Backup for GKE Architecture, Stateful Disaster Recovery, and Cross-Region Restoration

7 min read

Backup for GKE Architecture, Stateful Disaster Recovery, and Cross-Region Restoration 💾🛡️

Disaster recovery and compliance mandates require enterprise Kubernetes platforms to protect against accidental namespace deletions, ransomware data wiping, bad helm chart deployments, and regional datacenter failures. Backup for GKE is Google Cloud’s fully managed, cloud-native backup and disaster recovery service for Google Kubernetes Engine. Unlike complex, self-hosted third-party tools (like open-source Velero) that require maintaining S3 credentials, custom daemonsets, and fragile snapshot controllers, Backup for GKE provides a Google-managed control plane that orchestrates atomic snapshots of both Kubernetes resource configuration metadata (etcd objects) and underlying Persistent Volume data.

1. Architecture & The Backup Orchestration Pipeline

Backup for GKE operates via a managed GKE cluster add-on (gkebackup) that communicates securely with Google Cloud’s centralized backup service backplane.

                         GKE CLUSTER: PROD-US-CENTRAL1 (Source)
       ┌────────────────────────────────────────────────────────────────────────┐
       │   KUBERNETES CONTROL PLANE (ETCD)     WORKER NODE STATEFUL VOLUMES     │
       │   - Deployments, StatefulSets, Secrets│ - Persistent Volume Claims (PVCs)│
       │   - ConfigMaps, CRDs, RBAC bindings   │ - GCE Persistent Disks (Data)  │
       └───────────────────┬────────────────────────────────┬───────────────────┘
                           │                                │
                           ▼                                ▼
       ┌────────────────────────────────────────────────────────────────────────┐
       │                 BACKUP FOR GKE AGENT (gkebackup-agent)                 │
       │  - Quiesces stateful storage if pre-backup hook declared               │
       │  - Serializes all Kubernetes API manifests into compressed archives    │
       │  - Triggers GCE VolumeSnapshots via the Persistent Disk CSI driver     │
       └───────────────────┬────────────────────────────────┬───────────────────┘
                           │                                │
       ════════════════════╪════════════════════════════════╪════════════════════
       GOOGLE CLOUD BACKUP REPOSITORY (Immutable & Encrypted with CMEK)         │
                           │                                │
                           ▼                                ▼
       ┌────────────────────────────────────────────────────────────────────────┐
       │                     MANAGED BACKUP STORAGE VAULT                       │
       │                                                                        │
       │  ┌────────────────────────┐         ┌────────────────────────┐         │
       │  │ Resource Archive (.tar)│         │ Block Volume Snapshots │         │
       │  │ (etcd manifests & CRDs)│         │ (Incremental PD Deltas)│         │
       │  └────────────────────────┘         └────────────────────────┘         │
       └───────────────────────────────────┬────────────────────────────────────┘
                                           │ Cross-Region Restore Triggered
                                           ▼
                         GKE CLUSTER: PROD-EUROPE-WEST1 (Target DR)
       ┌────────────────────────────────────────────────────────────────────────┐
       │                 RESTORE FOR GKE AGENT (gkebackup-agent)                │
       │  - Remaps StorageClasses and Volume Types (e.g., Zonal PD ─► Regional) │
       │  - Recreates PVCs from storage snapshots                               │
       │  - Deploys stateful applications; resumes operations in minutes        │
       └────────────────────────────────────────────────────────────────────────┘

Core Architecture Constructs

  1. BackupPlan: A declarative template defining what to back up (entire cluster, specific namespaces, or label-selected workloads), the backup schedule (cron expression), retention policies, and encryption keys.
  2. Backup: The immutable, point-in-time artifact generated by a BackupPlan. Contains two synchronized components:
    • Kubernetes Resource Archive: YAML definitions of all selected workloads, secrets, service accounts, and CRDs.
    • Volume Data Snapshots: Underlying storage snapshots of all PersistentVolumeClaims bound to the backed-up pods.
  3. RestorePlan: A reusable blueprint defining restoration rules: destination cluster, conflict resolution policies (e.g., fail on conflict, rollback, or merge), volume remapping rules, and transformation rules.
  4. Restore: The execution of a RestorePlan restoring applications to a cluster.

2. Advanced Disaster Recovery: Cross-Region & Transformation Rules

During cross-region disaster recovery, differences in infrastructure frequently prevent raw restores:

  • StorageClass Substitution: A PVC backed by pd-ssd in us-central1 might need to restore onto hyperdisk-balanced in europe-west1. RestorePlans support native Volume Data Restoration Rules to substitute StorageClasses dynamically.
  • Substitution / Transformation Rules: Modify ingress hostnames, external IP references, or database connection strings dynamically during the restore process (e.g., rewriting prod-us.example.com to prod-eu.example.com).

3. Production Deployment & CLI Operations (gcloud)

1. Enable Backup for GKE Add-on on Cluster

# Enable Backup for GKE add-on
gcloud container clusters update prod-regional-cluster \
    --region=us-central1 \
    --update-addons=GkeBackup=ENABLED \
    --project=core-infrastructure-prod

2. Create a Production Automated BackupPlan with CMEK

Create a daily automated backup plan protecting the database and finance namespaces with 30-day retention:

gcloud beta container backup-restore backup-plans create daily-finance-backup-plan \
    --location=us-central1 \
    --cluster=projects/core-infrastructure-prod/locations/us-central1/clusters/prod-regional-cluster \
    --all-namespaces=false \
    --selected-namespaces=database,finance \
    --include-volume-data=true \
    --include-secrets=true \
    --cron-schedule="0 2 * * *" \
    --retention-policy-days=30 \
    --backup-delete-lock-days=7 \
    --encryption-key=projects/secops-kms-prod/locations/us-central1/keyRings/gke-backup-ring/cryptoKeys/backup-key \
    --project=core-infrastructure-prod

(Note: --backup-delete-lock-days=7 enforces immutability; backups cannot be deleted by anyone for 7 days, protecting against ransomware).

3. Trigger an Ad-Hoc On-Demand Backup

gcloud beta container backup-restore backups create manual-pre-migration-backup \
    --location=us-central1 \
    --backup-plan=daily-finance-backup-plan \
    --project=core-infrastructure-prod
 
# Monitor backup execution progress
gcloud beta container backup-restore backups describe manual-pre-migration-backup \
    --location=us-central1 \
    --backup-plan=daily-finance-backup-plan \
    --project=core-infrastructure-prod

4. Create a RestorePlan for Cross-Region Disaster Recovery

Deploy a RestorePlan targeting the secondary disaster-recovery cluster in europe-west1:

gcloud beta container backup-restore restore-plans create dr-restore-to-europe \
    --location=europe-west1 \
    --backup-plan=projects/core-infrastructure-prod/locations/us-central1/backupPlans/daily-finance-backup-plan \
    --cluster=projects/core-infrastructure-prod/locations/europe-west1/clusters/dr-cluster-eu \
    --namespaced-resource-restore-mode=RESTORE_ALL_NAMESPACED_RESOURCES \
    --volume-data-restore-policy=RESTORE_DATA_FROM_BACKUP \
    --cluster-resource-conflict-policy=USE_EXISTING_VERSION \
    --namespaced-resource-conflict-policy=FAIL_ON_CONFLICT \
    --project=core-infrastructure-prod

5. Execute Emergency Disaster Recovery Restoration

# Retrieve latest successful backup ID
LATEST_BACKUP=$(gcloud beta container backup-restore backups list \
    --location=us-central1 \
    --backup-plan=daily-finance-backup-plan \
    --sort-by=~createTime \
    --limit=1 \
    --format="value(name)")
 
# Trigger restoration into European DR cluster
gcloud beta container backup-restore restores create emergency-dr-restore-01 \
    --location=europe-west1 \
    --restore-plan=dr-restore-to-europe \
    --backup="${LATEST_BACKUP}" \
    --project=core-infrastructure-prod
 
# Verify restored pods in Europe cluster
kubectl get pods -n database --context=dr-cluster-eu

4. Quotas, Performance, and Configuration Limits

Dimension / ParameterLimit / BoundaryOperational Guidance
Max BackupPlans per Cluster100 plansOrganize plans by compliance tier or business service
Concurrent RestoresUp to 10 concurrentRestores scale with GCE disk provisioning throughput
Immutability Lock Window0 to 90 daysEnforce minimum 7 days for SOC2 / ransomware defense
Cross-Region RestoresSupported across all regionsRequires identical or remapped StorageClasses
Autopilot SupportFully supportedManaged agent deploys without privileged daemonset

5. Official References & Documentation

6. Realistic Pricing Scenarios

Pricing components:

  1. Backup Management Fee: Flat fee of $0.03 per Pod per month protected by an active backup plan.
  2. Persistent Volume Data Snapshots: Standard GCE VolumeSnapshot rates ($0.026 per GB-month).
  3. Cross-Region Network Egress: Billed only during cross-region restores ($0.02/GB).

Scenario A: Production Stateful Kubernetes Platform (50 Pods, 2 TB Storage)

  • Inventory:
    • 50 stateful and stateless pods across database and backend namespaces.
    • 10 Persistent Volume Claims totaling 2,000 GiB (2 TB) block storage.
    • Automated nightly backups retained for 30 days.
    • Average incremental snapshot delta accumulated: 2,500 GB.
  • Monthly Cost Calculation:
    • Backup Management Fee (50 pods): 50 pods × 1.50**
    • Snapshot Storage (2,500 GB): 2,500 GB × 65.00**
  • Total Monthly Cost: **70/month)._

Scenario B: Massive Multi-Tenant Enterprise GKE Cluster (1,000 Pods)

  • Inventory:
    • 1,000 pods protected across 20 namespaces.
    • 15 TB accumulated volume snapshot storage.
  • Monthly Cost Calculation:
    • Backup Management Fee: 1,000 pods × 30.00**
    • Snapshot Storage: 15,000 GB × 390.00**
  • Total Monthly Cost: $420.00 / month

7. Battle-Tested Nuggets & Production Gotchas

  1. The Inconsistent Database Snapshot (Pre-Backup Quiescing): If a high-write PostgreSQL or MySQL database is snapshotted while active transactions are in flight, restoring from that snapshot requires standard crash recovery. If write caching was dirty, the database may fail to start with index corruption. Use Kubernetes Pre-Backup and Post-Backup Hooks to run FLUSH TABLES WITH READ LOCK or SELECT pg_backup_start('gke_backup') immediately before the snapshot executes.
  2. Cross-Region StorageClass Mismatch Halts Restoration: When restoring a backup from us-central1 into europe-west1, if the destination cluster does not have the exact same StorageClass name declared (e.g., hyperdisk-balanced-encrypted), the restore controller cannot provision the PVCs. The restore will stall indefinitely in FAILED state. Always define StorageClass Mapping Rules in your RestorePlan to translate source storage classes into valid destination classes.
  3. Secret Encryption Keys Must Exist in Destination Region: If your BackupPlan encrypted the backup archives with a Cloud KMS key in us-central1, and you attempt to restore the backup into a cluster in europe-west1, the destination cluster service account must possess decryption permissions on that US KMS key. If cross-region IAM permissions are missing, the restore agent cannot decrypt the Kubernetes resource archive.
  4. Delete-Lock Protection Blocks Cluster Deletion: If you enable --backup-delete-lock-days=30, you cannot delete the BackupPlan or its backups for 30 days under any circumstances—even if you delete the source GKE cluster or attempt deletion with the Google Cloud Organization Admin account. Size delete-lock windows carefully in development environments to avoid orphan storage billing.
  5. Namespace Conflict Resolution Policy Pitfall: In a RestorePlan, if you configure --namespaced-resource-conflict-policy=FAIL_ON_CONFLICT, and a single ConfigMap or Secret already exists in the target namespace with the same name, the entire restore job halts immediately and rolls back. For disaster recovery into an existing cluster, configure USE_EXISTING_VERSION or REPLACE_EXISTING_VERSION.
  6. Local NVMe SSD Ephemeral Data Cannot Be Backed Up: Backup for GKE specifically protects persistent disks managed by the GCE CSI driver. Data residing in ephemeral storage (emptyDir), RAM disks, or physical Local NVMe SSDs is not captured in VolumeSnapshots. Ensure that all mission-critical state resides on persistent volume claims.

Across the wiki

Graph View

Backlinks