Cloud Pub/Sub Architecture & Streaming Mechanics 📨⚡

Google Cloud Pub/Sub is a globally distributed, horizontally scalable, multi-tenant asynchronous messaging middleware designed for 99.99% availability and massive throughput (millions of messages per second). Built on top of Google’s internal infrastructure (Colossus, Stubby/gRPC, and Borg), Pub/Sub completely decouples event producers from consumers without provisioning message brokers, partition rebalancing, or cluster capacity planning.

---

1. Global Architecture & Storage Mechanics

Unlike Apache Kafka or AWS Kinesis where topics are partitioned across specific broker nodes and regions, Pub/Sub separates the stateless routing frontend from the distributed message storage backend.

                        PUBLISHERS (Worldwide Microservices / IoT)
                                           │
                                           │ Global Anycast DNS / gRPC
                                           ▼
       ┌────────────────────────────────────────────────────────────────────────┐
       │                       PUB/SUB FORWARDING PROXIES                       │
       │                   (Global Anycast Edge Ingestion)                      │
       └───────────────────────────────────┬────────────────────────────────────┘
                                           │ Lowest Latency Region
                                           ▼
       ┌────────────────────────────────────────────────────────────────────────┐
       │                        PUB/SUB ROUTER FLEET                            │
       │  ┌────────────────────────┐         ┌────────────────────────┐         │
       │  │  Topic / Ordering Key  │         │  Schema Registry Check │         │
       │  │       Validator        │         │   (Avro / Protobuf)    │         │
       │  └───────────┬────────────┘         └───────────┬────────────┘         │
       └──────────────┼──────────────────────────────────┼──────────────────────┘
                      │                                  │
                      ▼                                  ▼
       ┌────────────────────────────────────────────────────────────────────────┐
       │                   DISTRIBUTED MESSAGE LOG STORAGE                      │
       │                                                                        │
       │   - Messages written synchronously to multiple availability zones      │
       │   - Colossus SSTable distributed log segments                          │
       │   - Automatic sharding based on message volume & ordering keys         │
       │   - Retention: 10 minutes to 31 days (Unacknowledged or Retained)       │
       └───────────────────────────────────┬────────────────────────────────────┘
                                           │ Pull / Push / Stream
                                           ▼
       ┌────────────────────────────────────────────────────────────────────────┐
       │                        PUB/SUB SUBSCRIBER FLEET                        │
       │                                                                        │
       │  ┌───────────────────────┐ ┌──────────────────────┐ ┌────────────────┐ │
       │  │ Pull / StreamingPull  │ │ Push / Cloud Run /   │ │ Direct Export  │ │
       │  │   (Ack/Nack Engine)   │ │ Webhook Endpoints    │ │ (BigQuery/GCS) │ │
       │  └───────────────────────┘ └──────────────────────┘ └────────────────┘ │
       └───────────────────────────────────┬────────────────────────────────────┘
                                           │
                        SUBSCRIBERS / DOWNSTREAM PIPELINES
                        (GKE, Cloud Run, Dataflow, BigQuery)

Core Ingestion & Storage Decoupling

1. Global Anycast Ingestion: Publishers connect to pubsub.googleapis.com via global Anycast IP routing. The connection terminates at the nearest Google edge PoP, which forwards traffic over Google’s private backbone to the publisher-designated region or the nearest available regional cluster.
2. Server-Driven Sharding: There is no concept of fixed partition counts. Pub/Sub automatically scales shards (called message partitions or lanes) dynamically up or down based on incoming write load and subscription processing rate.
3. Synchronous Quorum Durability: When a publisher receives an HTTP 200 / gRPC OK acknowledgment with a messageId, the message is already committed synchronously to disk across a quorum of independent zones.
4. Decoupled Subscription State: Topics do not store pointers for subscribers. Each Subscription maintains its own independent acknowledgment cursor and unacknowledged message queue. Adding 50 subscriptions to a topic has zero impact on topic publish throughput.

---

2. Deep Core Concepts & Engineering Mechanics

Message Lifecycle & Acknowledgment Mechanics

• Ack Deadline: When a message is delivered to a subscriber, the message’s ackDeadline timer begins (default: 10 seconds; maximum: 600 seconds).
• Lease Management (modifyAckDeadline): If message processing takes longer than the default deadline, client libraries (e.g., Google Cloud Java, Go, Python) automatically run a background lease extender thread that sends modifyAckDeadline requests to reset the timer until processing completes or the client crashes.
• Explicit NACK vs Timeout: Calling nack() immediately resets the ack deadline to 0, allowing another worker thread to pick up the message immediately rather than waiting for the timeout.

Ordering Keys & Delivery Guarantees

• Default Delivery: By default, Pub/Sub guarantees at-least-once delivery with unordered execution for maximum horizontal scaling across arbitrary workers.
• Message Ordering Keys: When a publisher sets an orderingKey (e.g., user_id_94821):
  • All messages sharing that ordering key are guaranteed to be delivered to consumers in the exact order they were published.
  • While one message for an ordering key is unacknowledged, subsequent messages for that key are blocked from delivery until the head-of-line message is ACKed or dead-lettered.
  • Ordering keys partition throughput: each ordering key can sustain up to ~1 MB/s (approx. 1,000 messages/sec).
• Exactly-Once Delivery: Enabled at the subscription level. Pub/Sub tracks acknowledgment tokens with distributed consensus. If an ACK is received, any redundant deliveries resulting from transient network retries are automatically suppressed by the service.

Dead-Letter Topics (DLT) & Exponential Backoff

• Max Delivery Attempts: Configured between 5 and 100. If a worker nacks or fails to acknowledge a message $N$ times, Pub/Sub automatically routes the poisoned payload to a designated Dead-Letter Topic.
• Retry Policy with Exponential Backoff: Enables consumers to configure minimum and maximum retry backoff delays (e.g., minimum 10s, maximum 600s) to prevent hammering downstream databases during transient outages.

Direct Ingestion Subscriptions (Zero-Code Pipelines)

Pub/Sub supports direct managed export subscriptions without needing intermediary Cloud Run or Dataflow workers:

• BigQuery Subscription: Streams incoming records directly into BigQuery tables with automatic schema detection and column matching.
• Cloud Storage Subscription: Batches messages and writes output files directly to GCS buckets based on file size (e.g., 50 MB) or time interval (e.g., 5 minutes) in Text, Avro, or Parquet formats.

---

3. Production Configuration & CLI (gcloud)

1. Create a Schema (Protobuf / Avro)

Create an Avro schema definition user_event.avsc:

{
  "type": "record",
  "name": "UserEvent",
  "namespace": "com.cloudnative.wiki",
  "fields": [
    { "name": "eventId", "type": "string" },
    { "name": "userId", "type": "string" },
    { "name": "eventType", "type": "string" },
    { "name": "timestamp", "type": "long" }
  ]
}

Register schema in the Pub/Sub schema registry:

gcloud pubsub schemas create user-event-schema \
    --type=AVRO \
    --definition-file=user_event.avsc \
    --project=telemetry-prod

2. Create Topic with Schema Validation & CMEK

gcloud pubsub topics create user-events-topic \
    --schema=user-event-schema \
    --message-encoding=JSON \
    --message-retention-duration=7d \
    --kms-key-name=projects/telemetry-prod/locations/us-central1/keyRings/pubsub-ring/cryptoKeys/pubsub-key \
    --project=telemetry-prod

3. Create Dead-Letter Topic

gcloud pubsub topics create user-events-deadletter \
    --message-retention-duration=14d \
    --project=telemetry-prod
 
# Grant Pub/Sub system identity publisher rights to the dead-letter topic
PROJECT_NUMBER=$(gcloud projects describe telemetry-prod --format='value(projectNumber)')
PUBSUB_SERVICE_ACCOUNT="service-${PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com"
 
gcloud pubsub topics add-iam-policy-binding user-events-deadletter \
    --member="serviceAccount:${PUBSUB_SERVICE_ACCOUNT}" \
    --role="roles/pubsub.publisher" \
    --project=telemetry-prod

4. Deploy High-Resilience Pull Subscription with Exactly-Once Delivery

gcloud pubsub subscriptions create user-events-worker-sub \
    --topic=user-events-topic \
    --ack-deadline=30 \
    --enable-exactly-once-delivery \
    --enable-message-ordering \
    --dead-letter-topic=user-events-deadletter \
    --max-delivery-attempts=5 \
    --min-retry-delay=10s \
    --max-retry-delay=300s \
    --message-retention-duration=7d \
    --project=telemetry-prod

5. Deploy Direct-to-BigQuery Subscription

# Grant Pub/Sub permissions to write to BigQuery
gcloud projects add-iam-policy-binding telemetry-prod \
    --member="serviceAccount:${PUBSUB_SERVICE_ACCOUNT}" \
    --role="roles/bigquery.dataEditor"
 
gcloud pubsub subscriptions create user-events-bigquery-sub \
    --topic=user-events-topic \
    --bigquery-table=telemetry-prod:analytics_warehouse.raw_events \
    --use-topic-schema \
    --write-metadata \
    --drop-unknown-fields \
    --project=telemetry-prod

6. Publish and Consume Test Messages

# Publish a compliant message with an ordering key
gcloud pubsub topics publish user-events-topic \
    --message='{"eventId":"evt-1001","userId":"usr-884","eventType":"LOGIN","timestamp":1714567890}' \
    --ordering-key="usr-884" \
    --project=telemetry-prod
 
# Pull message with auto-acknowledgment
gcloud pubsub subscriptions pull user-events-worker-sub \
    --auto-ack \
    --limit=1 \
    --project=telemetry-prod

---

4. Quotas, Performance, and Configuration Limits

Resource / Parameter	Default Quota	Maximum / Scalability Target
Topic Publish Throughput	100 MB/s per region	Millions of MB/s via quota request
Ordering Key Throughput	1 MB/s (~1,000 msg/sec)	Hard boundary per single ordering key
Message Size Limit	10 MB per message	Strict platform limit (use claim-check pattern for larger)
Ack Deadline Range	10 seconds default	10 seconds minimum to 600 seconds maximum
Message Retention	7 days default	10 minutes minimum to 31 days maximum
Max Delivery Attempts (DLT)	5 default	5 minimum to 100 maximum
Push Endpoint Timeout	10 seconds	Configurable up to 600 seconds
Subscriptions per Topic	10,000 subscriptions	Allows massive multi-tenant fan-out

---

5. Official References & Documentation

• Google Cloud Pub/Sub Documentation
• Pub/Sub Architectural Overview
• Pub/Sub Message Ordering Guide
• BigQuery Subscriptions Architecture
• Cloud Pub/Sub Pricing Matrix

---

6. Realistic Pricing Scenarios

Pub/Sub pricing is based primarily on:

1. Data Ingestion & Delivery: $40perTiB($0.04 per GB) for standard throughput.
   • First 10 GiB per month is free.
2. Direct Push/Pull Egress: Inter-region egress incurs standard GCP networking fees.
3. Storage (Seek & Unacknowledged Messages): $0.27 per GB-month for retained messages beyond default.

Scenario A: High-Throughput E-Commerce Event Bus

• Traffic Profile:
  • 500 million messages published per month.
  • Average message size: 2 KB.
  • Total Monthly Ingestion Volume: $500,000,000\times 2\text{ KB}=1,000,000,000\text{ KB}\approx 1,000\text{ GB}=1\text{ TB}$.
  • Fan-out: 3 Subscriptions (Inventory Worker, Fraud Detection Worker, BigQuery Export Sub).
  • Total Monthly Delivery Volume: $1\text{ TB}\times 3=3\text{ TB}$.
  • Total billable throughput: $1\text{ TB (publish)}+3\text{ TB (subscriber delivery)}=4\text{ TB}$.
• Monthly Cost Calculation:
  • Billable Volume: $4\text{ TB}\times 1,024\text{ GB/TB}=4,096\text{ GB}$.
  • Free Tier deduction: $4,096-10=4,086\text{ GB}$.
  • Throughput Cost: 4{,}086 \text{ GB} \times \0.04/\text{GB} = \mathbf{$163.44}$
• Total Monthly Cost: $163.44 / month

Scenario B: Massive Telemetry & IoT Fleet Ingestion

• Traffic Profile:
  • 10 billion IoT telemetry events per month.
  • Average message size: 500 bytes (rounded up to the minimum 1 KB billing increment per message).
  • Billable message size: 1 KB per message.
  • Ingestion Volume: $10,000,000,000\times 1\text{ KB}=10,000\text{ GB}\approx 9.76\text{ TiB}$.
  • Subscriptions: 1 Direct-to-GCS Subscription (batch archiver) + 1 StreamingPull Subscription (Dataflow real-time anomaly detection).
  • Total Delivery Volume: $2\times 9.76\text{ TiB}=19.52\text{ TiB}$.
  • Total billable volume: $9.76\text{ TiB (ingest)}+19.52\text{ TiB (delivery)}=29.28\text{ TiB}=30,000\text{ GB}$.
• Monthly Cost Calculation:
  • Throughput: 30{,}000 \text{ GB} \times \0.04/\text{GB} = \mathbf{$1{,}200.00}$
  • Snapshot / Seek retention storage (500 GB-month): 500 \times \0.27 = \mathbf{$135.00}$
• Total Monthly Cost: $1,335.00 / month

---

7. Battle-Tested Nuggets & Production Gotchas

1. The 1 KB Minimum Billing Boundary Trap: Pub/Sub bills message volume in minimum 1 KB increments per message. If your microservices publish 100-byte JSON payloads containing only { "status": "ok" }, you are billed for 1,000 bytes—a 10x cost multiplier. If you ingest high-velocity micro-events, batch multiple messages into a single array payload before publishing, or use Pub/Sub Lite / Kafka when payload sizes are minuscule.
2. Ordering Key Head-of-Line Blocking Disasters: When using orderingKey, if a single poisoned message causes an exception in your subscriber and fails to acknowledge, all subsequent messages for that ordering key are completely blocked from delivery. The ordering queue for that key halts indefinitely until the ack deadline expires repeatedly or the message is routed to a Dead-Letter Topic. Always configure a DeadLetterPolicy with maxDeliveryAttempts whenever enableMessageOrdering is enabled.
3. Dead-Letter Topics Require Explicit IAM Grant: Creating a subscription with --dead-letter-topic will fail silently or reject message forwarding unless the Google-managed Pub/Sub service account (service-<PROJECT_NUMBER>@gcp-sa-pubsub.iam.gserviceaccount.com) is explicitly granted roles/pubsub.publisher on the dead-letter topic and roles/pubsub.subscriber on the source subscription. If permissions are missing, poisoned messages are delivered repeatedly until the 7-day retention expires, exhausting subscriber CPU.
4. Client-Side Flow Control & Out-Of-Memory (OOM): The official Pub/Sub client libraries use asynchronous gRPC StreamingPull under the hood. By default, client libraries aggressively buffer hundreds of megabytes of messages in memory. If a downstream consumer slows down (e.g., waiting on slow SQL writes), the client process can experience sudden memory spikes and OOM crashes. Always configure strict flow control settings in code (e.g., FlowControlSettings.newBuilder().setMaxOutstandingElementCount(1000).setMaxOutstandingRequestBytes(50 * 1024 * 1024).build()).
5. Exactly-Once Delivery Adds Acknowledgment Latency: Enabling enable-exactly-once-delivery guarantees that retried network packets do not trigger duplicate execution. However, this relies on global Paxos/Spanner-backed acknowledgment token validation across Google’s distributed consensus layer. Standard subscriptions acknowledge in ~1-5ms; exactly-once subscriptions incur 20-50ms ACK response latencies and can encounter ALREADY_EXISTS or FAILED_PRECONDITION errors if client ack deadlines are configured too aggressively.
6. Cross-Region Network Egress Surcharge: Publishing to a topic from an AWS server or an on-premises datacenter via the public internet incurs standard Google Cloud external ingress (free) and egress (charged). However, if your GKE cluster resides in europe-west1 and pulls messages from a Pub/Sub topic where data storage was pinned to us-central1, you will incur silent inter-continental cross-region networking egress fees ($0.02-$0.08 per GB). Define explicit allowedPersistenceRegions in topic settings to keep message storage co-located with your compute clusters.

Across the wiki

• Amazon SQS — messaging and streaming (AWS)
• Azure Service Bus Architecture, Queues, Topics, and Enterprise Messaging — messaging and streaming (Azure)
• Apache Kafka — messaging and streaming (Architecture)
• Amazon SNS — messaging and streaming (AWS)