Cloud KMS, Cloud HSM, and CMEK Envelope Encryption 🔐🏛️

Google Cloud Key Management Service (Cloud KMS) is a fully managed, globally available cryptographic key management service that enables enterprises to create, manage, rotate, and control cryptographic keys used across Google Cloud workloads. Cloud KMS supports symmetric encryption, asymmetric encryption, digital signatures, hardware security modules (Cloud HSM validated to FIPS 140-2 Level 3), and External Key Management (Cloud EKM) to satisfy sovereign cloud and regulatory mandates.

---

1. Architecture & Envelope Encryption Model

By default, all data stored at rest within Google Cloud is encrypted using Google-managed encryption keys. Cloud KMS enables Customer-Managed Encryption Keys (CMEK), giving customers granular lifecycle control, auditability, and immediate cryptographic revocation capabilities over storage, databases, and compute disks.

                                  CLIENT / WORKLOAD
                                          │
            ┌─────────────────────────────┴─────────────────────────────┐
            │ 1. Request Data Encryption Key (DEK)                      │
            ▼                                                           │
    ┌────────────────────────────────────────────────────────────────┐  │
    │                    GOOGLE CLOUD KMS SERVICE                    │  │
    │                                                                │  │
    │  ┌──────────────────────────────────────────────────────────┐  │  │
    │  │                        KEY RING                          │  │  │
    │  │  (Regional or Multi-Regional Administrative Boundary)    │  │  │
    │  │                                                          │  │  │
    │  │  ┌────────────────────────────────────────────────────┐  │  │  │
    │  │  │                 CRYPTOKEY (KEK)                    │  │  │  │
    │  │  │  - Key Purpose: ENCRYPT_DECRYPT / ASYMMETRIC_SIGN  │  │  │  │
    │  │  │  - Protection Level: SOFTWARE / HSM / EXTERNAL     │  │  │  │
    │  │  │  - Auto-Rotation: e.g., every 90 days              │  │  │  │
    │  │  │  ┌──────────────────┐    ┌──────────────────┐      │  │  │  │
    │  │  │  │ CryptoKeyVersion │    │ CryptoKeyVersion │      │  │  │  │
    │  │  │  │   v1 (Destroyed) │    │   v2 (Primary)   │      │  │  │  │
    │  │  │  └──────────────────┘    └──────────────────┘      │  │  │  │
    │  │  └────────────────────────────────────────────────────┘  │  │  │
    │  └──────────────────────────────────────────────────────────┘  │  │
    └──────────────────────────────┬─────────────────────────────────┘  │
                                   │ 2. Returns Encrypted DEK           │
                                   ▼                                    │
    ┌────────────────────────────────────────────────────────────────┐  │
    │                     ENVELOPE ENCRYPTION FLOW                   │  │
    │                                                                │  │
    │  Plaintext Data + Plaintext DEK ──► AES-256-GCM ──► Ciphertext │◄─┘
    │                                                          ▲        │
    │                                                          │        │
    │  Encrypted DEK (Wrapped by KEK) ─────────────────────────┴────────┘
    │                                (Stored alongside ciphertext)
    └───────────────────────────────────────────────────────────────────┘

Core Cryptographic Mechanisms

1. Envelope Encryption: To minimize performance overhead and network latency, large datasets are never streamed directly through the Cloud KMS API. Instead:
   • A local Data Encryption Key (DEK) is generated in memory using AES-256-GCM.
   • The plaintext data is encrypted locally with the DEK.
   • The DEK is encrypted (“wrapped”) by the Key Encryption Key (KEK) residing securely inside Cloud KMS.
   • The wrapped DEK is stored alongside the ciphertext. Plaintext DEK is wiped from RAM immediately.
2. Protection Levels:
   • Software: High-speed software-backed keys running in Google’s secure microkernel environment.
   • Cloud HSM: Dedicated hardware security module clusters certified to FIPS 140-2 Level 3. Keys can never be extracted in plaintext from the HSM boundary.
   • Cloud EKM (External Key Manager): Keys are stored and managed inside an on-premises or third-party HSM outside of Google’s datacenters (e.g., Thales CipherTrust, Fortanix). Google Cloud calls your external EKM over an authenticated TLS/PSC tunnel for each cryptographic operation.

---

2. Key Hierarchy & Lifecycle Management

Hierarchy Taxonomy

• Project: The overarching GCP resource containing billing, IAM policies, and VPC configuration.
• Location: Key rings are regional (e.g., us-central1), dual-regional (nam4), or global (global). Best practice is to co-locate the Key Ring in the exact same region as the encrypted resources to minimize cross-region latency and eliminate external dependency risks during regional outages.
• Key Ring: An administrative container that organizes keys and centralizes IAM permissions. Key rings cannot be deleted once created (to preserve non-repudiation audit trails).
• CryptoKey: The named object representing the logical key. Defines the cryptographic algorithm, purpose, protection level, and rotation period.
• CryptoKeyVersion: The physical cryptographic key material. A key can have multiple versions. One version is designated PRIMARY for encryption, while older versions remain active for decryption of historical data.

Key Lifecycle States

A CryptoKeyVersion transitions through strict deterministic lifecycle states:

1. PENDING_GENERATION $\to$ ENABLED (Active and usable).
2. DISABLED (Temporarily suspended; cryptographic operations fail immediately).
3. DESTROY_SCHEDULED (Marked for destruction with a mandatory waiting period, typically 24 hours to 30 days, to protect against accidental catastrophic data loss).
4. DESTROYED (Key material permanently erased from physical media and HSM memory; data encrypted with this version is permanently irrecoverable).

---

3. Production Deployment & CLI Operations (gcloud)

1. Create a Regional Key Ring and HSM CryptoKey

# Create regional key ring
gcloud kms keyrings create prod-security-ring \
    --location=us-central1 \
    --project=secops-kms-prod
 
# Create AES-256 symmetric key protected by FIPS 140-2 Level 3 Hardware HSM
gcloud kms keys create gcs-storage-key \
    --keyring=prod-security-ring \
    --location=us-central1 \
    --purpose=encryption \
    --protection-level=hsm \
    --rotation-period=90d \
    --next-rotation-time="2026-10-01T00:00:00Z" \
    --project=secops-kms-prod

2. Grant CMEK Access to Google Managed Service Accounts

When integrating CMEK with GCS, BigQuery, Compute Engine, or Cloud SQL, the respective service agent must be granted the roles/cloudkms.cryptoKeyEncrypterDecrypter role on the specific key.

# Retrieve the Google Cloud Storage system service account
STORAGE_SA=$(gcloud storage service-agent --project=data-storage-prod)
 
# Grant Encrypter/Decrypter permissions to GCS service agent
gcloud kms keys add-iam-policy-binding gcs-storage-key \
    --keyring=prod-security-ring \
    --location=us-central1 \
    --member="serviceAccount:${STORAGE_SA}" \
    --role="roles/cloudkms.cryptoKeyEncrypterDecrypter" \
    --project=secops-kms-prod

3. Deploy CMEK-Protected Cloud Storage Bucket

gcloud storage buckets create gs://finance-ledger-prod \
    --location=us-central1 \
    --default-encryption-key=projects/secops-kms-prod/locations/us-central1/keyRings/prod-security-ring/cryptoKeys/gcs-storage-key \
    --uniform-bucket-level-access \
    --project=data-storage-prod

4. Create an Asymmetric Key for Digital Signatures (RSA-PSS 4096)

gcloud kms keys create prod-code-signing-key \
    --keyring=prod-security-ring \
    --location=us-central1 \
    --purpose=asymmetric-signing \
    --default-algorithm=rsa-sign-pss-4096-sha512 \
    --protection-level=hsm \
    --project=secops-kms-prod
 
# Download the public key for verification by external verifiers
gcloud kms keys versions get-public-key 1 \
    --key=prod-code-signing-key \
    --keyring=prod-security-ring \
    --location=us-central1 \
    --output-file=signing-pubkey.pem \
    --project=secops-kms-prod

5. Emergency Key Revocation & Restoration

# Emergency Disable Key Version (Halts all CMEK reads/writes across databases immediately)
gcloud kms keys versions disable 1 \
    --key=gcs-storage-key \
    --keyring=prod-security-ring \
    --location=us-central1 \
    --project=secops-kms-prod
 
# Re-enable Key Version after incident resolution
gcloud kms keys versions enable 1 \
    --key=gcs-storage-key \
    --keyring=prod-security-ring \
    --location=us-central1 \
    --project=secops-kms-prod
 
# Schedule Version for permanent destruction with safety buffer
gcloud kms keys versions destroy 1 \
    --key=gcs-storage-key \
    --keyring=prod-security-ring \
    --location=us-central1 \
    --project=secops-kms-prod

---

4. Quotas, Performance, and Configuration Limits

Parameter / Resource	Default Limit	Engineering Guidance
Symmetric Crypto Operations	60,000 QPS per region	Software keys scale virtually without limit
Cloud HSM Operations	3,000 QPS per region	Suitable for envelope KEK operations, not raw data streams
Asymmetric Sign / Decrypt	300 QPS per region	Cache public keys locally; only verify client-side
Key Rings per Location	Unlimited	Cannot be deleted once created
CryptoKeys per Key Ring	Unlimited	Group keys by compliance tier or business service
Key Rotation Frequency	Min 24 hours	Recommended enterprise standard: 90 to 365 days
Destruction Scheduled Delay	24 hours to 120 days	Default is 24 hours; set to 30 days for production safety

---

5. Official References & Documentation

• Google Cloud KMS Documentation
• Envelope Encryption Architecture
• Cloud HSM Architecture & FIPS Validation
• Cloud External Key Manager (Cloud EKM)
• Cloud KMS Pricing Guide

---

6. Realistic Pricing Scenarios

Cloud KMS pricing is based on:

1. Active Key Versions:
   • Software Key Version: $0.06 per key version / month.
   • Cloud HSM Key Version: $1.00 per key version / month.
   • Cloud EKM Key Version: $2.00 per key version / month.
2. Cryptographic Operations:
   • Software Symmetric Operations: $0.03 per 10,000 operations.
   • Cloud HSM Operations: $0.03 per 10,000 operations.
   • Asymmetric Operations: $0.03 per 10,000 operations.

Scenario A: Standard Enterprise Cloud Infrastructure (Software CMEK)

• Inventory:
  • 100 GCS Buckets, 20 Persistent Disks, 10 BigQuery Datasets.
  • 50 CryptoKeys (Software-backed), rotated annually (average 1.5 active versions per key = 75 active versions).
  • API Operations: Envelope encryption means keys are only invoked when instances mount disks, buckets initialize, or batch jobs run (~500,000 operations/month).
• Monthly Cost Calculation:
  • Key Version Storage: 75 versions × $0.06=\ast \ast$4.50**
  • Cryptographic Operations: (500{,}000 / 10{,}000) \times \0.03 = \mathbf{$1.50}$
• Total Monthly Cost: $6.00 / month

Scenario B: Highly Regulated Financial Banking Platform (Cloud HSM + High QPS)

• Inventory:
  • 200 Cloud HSM Keys (FIPS 140-2 Level 3) for transaction signing, tokenization, and database CMEK.
  • Active Versions: 200 keys × 2 versions = 400 active HSM versions.
  • API Operations: Microservice envelope caching decrypts DEKs on service restart; tokenization service performs 15 million HSM cryptographic operations per month.
• Monthly Cost Calculation:
  • HSM Key Versions: 400 versions × $1.00=\ast \ast$400.00**
  • HSM Operations: (15{,}000{,}000 / 10{,}000) \times \0.03 = \mathbf{$45.00}$
• Total Monthly Cost: $445.00 / month

---

7. Battle-Tested Nuggets & Production Gotchas

1. Key Rings and Keys Cannot Be Deleted: In Google Cloud, gcloud kms keyrings delete or gcloud kms keys delete does not exist. Once created, a Key Ring and Key exist forever to maintain cryptographic non-repudiation and prevent audit log tampering. Only individual CryptoKeyVersion entries can have their key material destroyed. Always use consistent naming conventions (<service>-<environment>-key) and avoid creating temporary test keys with production names.
2. Key Rotation Does NOT Re-Encrypt Existing Data: When a key auto-rotates (e.g., version 1 rotates to version 2), version 2 becomes the PRIMARY version used to encrypt new data. It does not automatically re-encrypt historical tables, GCS blobs, or persistent disks created with version 1. To completely decommission version 1, you must perform a batch rewrite of historical data (e.g., copying GCS objects in place or running ALTER TABLE in BigQuery) before scheduling version 1 for destruction.
3. Cross-Region CMEK Dependency Failure Mode: Never configure a resource in europe-west1 to use a CMEK key located in us-central1. If an undersea fiber disruption or regional control-plane degradation affects us-central1, services in europe-west1 will fail to decrypt their DEKs, causing persistent disks to detach and Cloud SQL instances to crash into recovery mode. Always keep KMS keys co-located in the same region as the data they protect.
4. Disabling a CMEK Key Instantly Kills Compute & Databases: If an administrator disables or destroys a CMEK key used by a Compute Engine boot disk, GKE node pool, or Cloud SQL database, the virtual machines are halted or enter an error state within 15 to 30 minutes when memory leases expire. Disabling a key is the ultimate cryptographic “kill switch,” but it must be guarded by strict IAM deny policies and Break-Glass procedures.
5. Separation of Duties (Project-Level Isolation): Never store Cloud KMS keys in the same GCP project as the workloads consuming them. If an attacker compromises a project owner credential in app-production, they could grant themselves decryption rights or delete keys. Place KMS in a dedicated secops-kms-prod project where only security administrators have IAM admin privileges, granting application service accounts only the granular roles/cloudkms.cryptoKeyEncrypterDecrypter role.
6. Cloud HSM vs Cloud EKM Latency Realities: While Cloud HSM delivers sub-5 millisecond response times inside Google’s datacenters, Cloud EKM forwards cryptographic requests over external internet or Interconnect connections to an external on-premises HSM. If your external HSM or internet connection experiences latency or jitter, all Cloud Storage uploads and database queries will experience severe latency degradation or timeout errors. Implement client-side DEK caching whenever possible.

Across the wiki

• AWS KMS — key management (AWS)
• Azure Key Vault Architecture, Managed HSM, and Cryptographic Governance — key management (Azure)
• Cryptography — key management (Architecture)
• etcd Encryption — key management (Kubernetes)