cloudnative wiki

GCP Application Integration & Eventing Hub

3 min read

GCP Application Integration & Eventing Hub 🔌

Modern cloud-native architectures require decoupled asynchronous communication, event-driven triggers, durable workflow state machines, and enterprise API gateways. This hub details Google Cloud’s application integration portfolio, enabling resilient microservice topologies and event routing across cloud boundaries.

Service Catalog & Architecture Map

ServiceArchitecture & Protocol PatternPrimary Use Case & GuaranteesReference Guide
Cloud TasksAsynchronous HTTP Dispatch / Push QueuePoint-to-point task queueing, token-bucket rate limiting, automatic backoff retries, and task deduplication.Cloud Tasks Deep Dive
WorkflowsServerless State Machine / JSON & YAML DAGLow-latency microservice orchestration, long-running stateful workflows (up to 1 year), and built-in connector retries.Workflows Deep Dive
EventarcCloudEvents Specification Event RouterStandardized event delivery routing audit logs, GCS mutations, and Pub/Sub events directly to Cloud Run, GKE, or Workflows.Eventarc Deep Dive
Apigee XEnterprise API Management & GatewayFull lifecycle API governance, traffic shaping (spike arrest/quotas), developer portal, monetization, and Cloud Armor security.Apigee X Deep Dive
Cloud Pub/SubHigh-Throughput Distributed LogGlobal Anycast pub/sub messaging engine supporting millions of messages/sec with ordering keys and dead-letter topics.Sub Deep Dive
Cloud ComposerManaged Apache Airflow OrchestrationData workflow pipeline orchestration, complex DAG scheduling, and native BigQuery/Dataflow/Spark connectors.Cloud Composer Deep Dive

Architectural Decision Framework

flowchart TD
    Req["Integration Requirement"] --> Pattern{"Communication Pattern & Execution Model?"}

    Pattern -->|"High-Volume Stream / Many-to-Many"| PubSub["Use Cloud Pub/Sub (Global Anycast, Fan-Out)"]
    Pattern -->|"Point-to-Point Task Queue / Rate Limited"| Tasks["Use Cloud Tasks (Target Dispatch, Token Bucket)"]
    Pattern -->|"Standard CloudEvents from GCP Services"| Eventarc["Use Eventarc (GCS/Audit Log triggers)"]
    Pattern -->|"Multi-Step Conditional Business Logic"| Orchestration{"What type of pipeline?"}
    Pattern -->|"External Partner APIs / Monetization"| Apigee["Use Apigee X (Enterprise Gateway, Developer Portal)"]

    Orchestration -->|"Microservice APIs / Sub-Second Serverless"| Workflows["Use Workflows (JSON/YAML State Machine)"]
    Orchestration -->|"Data Engineering / Python DAGs / Airflow"| Composer["Use Cloud Composer (Apache Airflow on GKE)"]

1. Cloud Tasks vs Cloud Pub/Sub

  • Cloud Pub/Sub (Publisher-Driven): Engineered for many-to-many broadcast streaming. The publisher has no knowledge of downstream consumers. Does not support explicit dispatch rate throttling per consumer or point-in-time future scheduling.
  • Cloud Tasks (Consumer-Driven): Engineered for one-to-one task dispatch. The queue explicitly dictates the execution target (e.g. POST https://orders-api.internal/process), enforces strict concurrency limits (e.g. maximum 10 concurrent requests), and supports scheduling tasks up to 30 days in the future.

2. Workflows vs Cloud Composer

  • Workflows: Lightweight, serverless, fast cold-start (sub-millisecond execution overhead), billed per step executed ($0.00 for idle). Ideal for user-facing API orchestration, payment processing sagas, and serverless event handlers.
  • Cloud Composer: Heavyweight, persistent Apache Airflow environment with dedicated schedulers, Celery workers, and web server. Ideal for complex batch ETL pipelines, data lake ingestion, machine learning workflows, and dependency graphs with complex Python dependencies.
  • GCP — Master Google Cloud Platform architectural catalog
  • Cloud Run — Primary execution runtime for serverless event handlers and API endpoints
  • Sub — Global messaging backbone
  • Event-Driven Architecture — Core patterns: Saga, Outbox, Choreography vs Orchestration

Graph View

Backlinks