AKS Backup, Disaster Recovery, and Cross-Region Business Continuity 🛡️🌍

Disasters in cloud-native environments stem from multiple failure vectors: accidental kubectl delete namespace commands by human operators, corrupt Helm releases, regional Azure fiber cuts, or ransomware attacks. Architecting enterprise resilience on Azure Kubernetes Service (AKS) requires a dual-track strategy: Data Protection via Azure Backup for AKS (a fully managed, enterprise distribution of open-source Velero) to backup stateful volumes and Kubernetes manifests, paired with Multi-Region Active-Active Traffic Routing via Azure Front Door to achieve near-zero Recovery Time Objectives (RTO).

---

1. Architecture: The AKS Backup & Recovery Control Plane

Azure Backup for AKS deploys an extension into the cluster that integrates the Kubernetes control plane directly with an Azure Backup Vault and a target Azure Storage Account (Blob Storage).

       ┌────────────────────────────────────────────────────────────────────────┐
       │ PRIMARY AKS REGION: `eastus`                                           │
       │                                                                        │
       │  ┌────────────────────────┐         ┌────────────────────────┐         │
       │  │ Cluster Resources (K8s)│         │ Stateful Disks (PVCs)  │         │
       │  │ - Namespaces, ConfigMap│         │ - PostgreSQL 500 GiB   │         │
       │  │ - Deployments, Secrets │         │ - VolumeSnapshots (CSI)│         │
       │  └───────────┬────────────┘         └───────────┬────────────┘         │
       │              │ Synchronizes Manifests           │ Incremental Deltas   │
       │              ▼                                  ▼                      │
       │  ┌──────────────────────────────────────────────────────────┐          │
       │  │ AZURE BACKUP EXTENSION FOR AKS (Managed Velero Core)     │          │
       │  │ - Scheduled BackupPolicy (Daily at 02:00 UTC)            │          │
       │  │ - Workload Identity Authentication                       │          │
       │  └───────────────────────────┬──────────────────────────────┘          │
       └──────────────────────────────┼─────────────────────────────────────────┘
                                      │ Encrypted Cross-Region Replication (GRS)
                                      ▼
       ┌────────────────────────────────────────────────────────────────────────┐
       │ AZURE BACKUP VAULT & GRS STORAGE ACCOUNT (Secondary Region: `westus`)  │
       │                                                                        │
       │  - Point-in-time immutable backup snapshots                            │
       │  - Zero-data-loss cross-region replication (RPO < 15 minutes)          │
       └──────────────────────────────────┬─────────────────────────────────────┘
                                          │ One-Click Restore / Automated Failover
                                          ▼
       ┌────────────────────────────────────────────────────────────────────────┐
       │ SECONDARY AKS REGION: `westus` (STANDBY / RESTORE TARGET)              │
       │  - Restores all PVCs from snapshot metadata                            │
       │  - Recreates Deployments, Services, and Ingress routing                │
       └────────────────────────────────────────────────────────────────────────┘

---

2. Disaster Recovery Strategy Comparison

Strategy	Recovery Time Objective (RTO)	Recovery Point Objective (RPO)	Cost Overhead	Complexity
Backup & Restore (Cold Standby)	1 to 4 hours	1 to 24 hours (Snapshot age)	Lowest (~5% overhead)	Low
Pilot Light (Warm Standby)	15 to 30 minutes	< 1 hour	Moderate (~30% overhead)	Moderate
Active-Passive Failover	< 5 minutes	Near Zero (Database replication)	High (~100% overhead)	High
Active-Active Multi-Region	< 10 seconds (Instant)	Zero (Synchronous multi-region)	Highest (> 200% overhead)	Advanced

---

3. Production Deployment & CLI Operations (az CLI)

1. Provision Backup Vault and Storage Account

# Register required Azure Backup providers
az provider register --namespace Microsoft.DataProtection
 
# Create Resource Group for centralized backup infrastructure
az group create --name rg-prod-backup --location eastus
 
# Create an Azure Backup Vault with Cross-Region Restore (CRR) enabled
az dataprotection backup-vault create \
    --resource-group rg-prod-backup \
    --vault-name bvault-aks-prod \
    --location eastus \
    --storage-settings datastore-type="VaultStore" type="GeoRedundant" \
    --cross-region-restore-state "Enabled"
 
# Create a dedicated Storage Account to store Kubernetes manifest blobs
az storage account create \
    --name staksprodbackups \
    --resource-group rg-prod-backup \
    --location eastus \
    --sku Standard_GRS

2. Install Azure Backup Extension on the AKS Cluster

# Install the managed Backup extension on AKS
az k8s-extension create \
    --name azure-aks-backup \
    --extension-type Microsoft.DataProtection.Kubernetes \
    --scope cluster \
    --cluster-type managedClusters \
    --cluster-name aks-core-prod \
    --resource-group rg-prod-aks \
    --configuration-settings \
        blobContainer="aks-manifest-backups" \
        storageAccount="staksprodbackups" \
        storageAccountResourceGroup="rg-prod-backup" \
        storageAccountSubscription="00000000-0000-0000-0000-000000000000"

3. Create Backup Policy and Schedule Daily Backups

# Create Backup Policy JSON defining retention of 30 days
cat <<EOF > backup-policy.json
{
  "policyRules": [
    {
      "backupParameters": {
        "backupType": "Incremental"
      },
      "trigger": {
        "schedule": {
          "repeatingTimeIntervals": [ "R/2026-09-06T02:00:00+00:00/P1D" ]
        },
        "triggerType": "ScheduleBasedTriggerContext"
      },
      "dataStore": {
        "dataStoreType": "OperationalStore",
        "objectType": "OperationalDataStoreSettings"
      },
      "name": "DailyBackupRule",
      "objectType": "AzureBackupRule"
    }
  ],
  "name": "AksDailyRetentionPolicy",
  "objectType": "BackupPolicy"
}
EOF
 
# Create backup policy in the Backup Vault
az dataprotection backup-policy create \
    --resource-group rg-prod-backup \
    --vault-name bvault-aks-prod \
    --name AksDailyPolicy \
    --policy backup-policy.json

4. Restore Cluster Workloads during an Incident

In the event of accidental deletion or regional failover, restore the production namespace to the standby cluster:

# Trigger an automated restore operation targeting the secondary cluster
az dataprotection backup-instance restore trigger \
    --resource-group rg-prod-backup \
    --vault-name bvault-aks-prod \
    --backup-instance-name "aks-core-prod-instance" \
    --restore-request-object restore-request.json

---

4. Quotas, Performance & Configuration Limits

Parameter / Capability	Platform Limit	Production Context
Max Backup Instances per Vault	1,000 Clusters	Centralized enterprise governance
Snapshot Consistency	Crash-Consistent	Pre-freeze/post-thaw hooks required for DB ACID
Supported Storage Drivers	Azure Disk CSI & Azure Files CSI	Backs up both block disks and shared files
Cross-Region Restore (CRR)	Supported via GRS	Allows restoring directly to paired secondary region
Max Backup Retention	Up to 10 Years	Satisfies regulatory compliance (HIPAA/FINRA)

---

5. Official References

• Azure Backup for AKS Overview
• Disaster Recovery Best Practices for AKS
• Velero Open Source Project
• Azure Backup Pricing

---

6. Realistic Pricing Scenarios

Scenario A: Enterprise Stateful Workload Backup (5 Clusters, 20 Disks)

• Architecture: 5 production AKS clusters running 20x 500 GiB Azure Managed Disks (10 TiB total state).
• Policy: Daily snapshots with 30-day retention stored in GRS Storage.
• Monthly Cost Breakdown:
  • Protected Instances Fee: 5 clusters × $40.00/instance/month=\ast \ast$200.00**
  • GRS Blob Storage (10 TiB + 20% incremental deltas = 12 TiB): 12,288 GiB × $0.036/GiB=\ast \ast$442.36**
  • Snapshot Restore Operations: ~$10.00
• Total Monthly Disaster Recovery Cost: $652.36 / month

Scenario B: Multi-Region Active-Active Front Door Ingress

• Architecture: 2 identical 10-node AKS clusters (East US and West Europe) fronted by Azure Front Door Premium with Anycast DNS health probing.
• Monthly Cost Breakdown:
  • Primary & Secondary Compute (20x D8ds_v5): 20 × $0.384/hr\times 730hrs=\ast \ast$5,606.40**
  • Azure Front Door Premium Base: $330.00 / month
  • Cross-Region Data Transfer (50 TB): 50,000 GB × $0.08/GB=\ast \ast$4,000.00**
  • Standard Control Plane SLAs (2 clusters): 2 × $73.00=\ast \ast$146.00**
• Total Active-Active Monthly Spend: $10,082.40 / month (Delivering sub-10 second automated global failover).

---

7. Battle-Tested Nuggets & Production Gotchas

1. Crash-Consistent vs. Application-Consistent Snapshots: By default, CSI VolumeSnapshots are crash-consistent, capturing disk state while the database is actively writing. If PostgreSQL or MongoDB is in the middle of committing an un-flushed transaction buffer, restoring the snapshot will require database journal recovery, which can occasionally lead to database corruption. Configure pre-backup and post-backup hooks in your BackupPolicy to execute pg_start_backup() and pg_stop_backup().
2. Missing Storage Account Firewall Exceptions: If your backup Storage Account has its Azure Firewall enabled, the AKS Backup Extension pods will fail to upload manifests with 403 Forbidden. You must enable the Storage Account option "Allow Azure services on the trusted services list to access this storage account" and authorize the extension’s Managed Identity.
3. Cross-Region PVC StorageClass Compatibility: When restoring a backup from East US into West US, if your PVC references a StorageClass configured with a local zonal SKU (e.g., PremiumV2_LRS or custom UltraDisk flags) that is unavailable in the target region’s secondary datacenter, the restore operation will stall with ProvisioningFailed: StorageClass not found. Ensure StorageClasses in the standby cluster have identical naming and configuration.
4. Active-Active Split-Brain Database Collisions: Deploying identical microservices in two active regions fronted by Azure Front Door is straightforward for stateless code, but catastrophic for relational state. If both regions attempt to write simultaneously to single-master Azure SQL or PostgreSQL without distributed conflict resolution, data becomes irrevocably corrupted. Use active-passive failover for databases or adopt globally distributed multi-master stores like Azure Cosmos DB.
5. Backup Extension Workload Identity Permission Drops: When re-imaging nodes or rotating Managed Identity credentials, if the Backup Extension’s federated credential is deleted or expires, backup jobs will silently begin failing. Configure Azure Monitor Action Groups to fire high-priority Slack/PagerDuty alerts whenever BackupJobStatus != "Completed".

Across the wiki

• Backup for GKE Architecture, Stateful Disaster Recovery, and Cross-Region Restoration — backup and disaster recovery (GCP)
• Backup & Restore — backup and disaster recovery (Kubernetes)
• Disaster Recovery — backup and disaster recovery (Architecture)
• 2.2 Business Continuity — backup and disaster recovery (AWS)