cloudnative wiki

AWS Distro for OpenTelemetry on EKS

4 min read

AWS Distro for OpenTelemetry on EKS

AWS Distro for OpenTelemetry (ADOT) is AWS’s supported build of the OpenTelemetry Collector and SDKs. It is upstream OpenTelemetry with the AWS components included and tested: SigV4 authentication, exporters for X-Ray and CloudWatch, resource detectors for EKS and EC2.

Use it when you want one agent and one instrumentation standard for traces, metrics and logs, with the freedom to change backends later. If the OpenTelemetry model is new, read OpenTelemetry overview and the Collector first — this page covers only the EKS-specific parts.

What you install

The EKS add-on named adot installs the OpenTelemetry Operator, not a collector. The operator gives you two custom resources:

ResourcePurpose
OpenTelemetryCollectorDeclares a collector: its mode, image and pipeline configuration
InstrumentationDeclares how to inject auto-instrumentation agents into pods without changing their images
# The operator's webhooks need certificates
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml
 
aws eks create-addon --cluster-name my-cluster --addon-name adot

Collector deployment modes

ModeTopologyUse for
daemonsetOne collector per nodeNode and kubelet metrics, container logs, a local OTLP endpoint for applications
deploymentA scalable central poolGateway: batching, tail sampling, cluster-level metrics, fan-out to several backends
statefulsetStable identitiesPrometheus scraping with the target allocator, so each target is scraped exactly once
sidecarOne collector per podFargate, or strict per-tenant isolation

A production layout usually has two tiers: a DaemonSet agent that receives from local pods and forwards, and a Deployment gateway that holds the credentials and export logic. Applications then send to http://$(HOST_IP):4317 and never know about the backend.

A complete pipeline

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: gateway
  namespace: observability
spec:
  mode: deployment
  replicas: 2
  serviceAccount: adot-collector
  config:
    extensions:
      sigv4auth:
        region: eu-west-1
        service: aps
    receivers:
      otlp:
        protocols:
          grpc: { endpoint: 0.0.0.0:4317 }
          http: { endpoint: 0.0.0.0:4318 }
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 80
        spike_limit_percentage: 20
      k8sattributes: {}
      resourcedetection:
        detectors: [env, eks, ec2]
      batch: {}
    exporters:
      prometheusremotewrite:
        endpoint: https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/ws-1234/api/v1/remote_write
        auth: { authenticator: sigv4auth }
      awsxray:
        region: eu-west-1
      awscloudwatchlogs:
        region: eu-west-1
        log_group_name: /eks/my-cluster/otel
        log_stream_name: app
    service:
      extensions: [sigv4auth]
      pipelines:
        metrics:
          receivers: [otlp]
          processors: [memory_limiter, k8sattributes, resourcedetection, batch]
          exporters: [prometheusremotewrite]
        traces:
          receivers: [otlp]
          processors: [memory_limiter, k8sattributes, resourcedetection, batch]
          exporters: [awsxray]
        logs:
          receivers: [otlp]
          processors: [memory_limiter, k8sattributes, batch]
          exporters: [awscloudwatchlogs]

Order matters in processors: memory_limiter first so the collector sheds load before it is OOM-killed, batch last.

Permissions

Bind the collector’s service account to a role with Pod Identity or IRSA:

BackendManaged policy
Managed PrometheusAmazonPrometheusRemoteWriteAccess
X-RayAWSXrayWriteOnlyAccess
CloudWatch metrics and logsCloudWatchAgentServerPolicy

The k8sattributes processor also needs Kubernetes RBAC to get, list and watch pods, namespaces and replica sets.

Auto-instrumentation

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: default
  namespace: payments
spec:
  exporter:
    endpoint: http://gateway-collector.observability:4317
  propagators: [tracecontext, baggage, xray]
  sampler:
    type: parentbased_traceidratio
    argument: "0.1"

Annotate a workload’s pod template with instrumentation.opentelemetry.io/inject-java: "true" (or -python, -nodejs, -dotnet) and the operator adds an init container that loads the agent. This gives traces and runtime metrics for common frameworks with no code change. Include the xray propagator if requests pass through AWS services that only understand the X-Ray header — details in context propagation.

ADOT or the CloudWatch agent

The CloudWatch Observability add-on is the turnkey alternative: it installs the CloudWatch agent with Container Insights and Application Signals preconfigured. Choose it when CloudWatch is the only backend and you want the least configuration. Choose ADOT when you want vendor-neutral pipelines, Prometheus-compatible storage, or to send the same telemetry to more than one place.

Operating it

  • The collector exposes its own metrics on port 8888. Alert on otelcol_exporter_send_failed_* and otelcol_processor_refused_*; they are the first sign of dropped telemetry.
  • Enable the debug exporter temporarily to see exactly what a pipeline emits.
  • High-cardinality attributes become high-cardinality Prometheus labels and a large bill. Drop them in the collector with the attributes or transform processor.
  • Pin the add-on version and read release notes: collector configuration keys do change between versions.

Graph View

Backlinks