AWS Secrets Manager with EKS

Kubernetes Secrets are base64-encoded objects in etcd. They are fine as a delivery mechanism and poor as a system of record: no rotation, no versioning, coarse access control. The usual EKS pattern keeps the source of truth in AWS Secrets Manager (or SSM Parameter Store) and delivers values to pods at runtime.

There are two mainstream ways to do that, and they differ in where the secret ends up.

Two delivery models

	Secrets Store CSI Driver + AWS provider (ASCP)	External Secrets Operator (ESO)
Secret reaches the pod as	Files on a tmpfs volume	A normal Kubernetes Secret (env vars or volume)
Stored in etcd	No, unless you opt into sync	Yes
Who calls AWS	The node DaemonSet, using the pod’s identity	A central controller, using its own or a per-store role
Works when AWS API is down	Running pods keep files; new pods cannot start	Yes, pods start from the cached Secret
Application change needed	Read from a file	None
Fargate	No (needs a DaemonSet)	Yes

Choose the CSI driver when the requirement is “secrets never rest in etcd” and per-pod access control. Choose ESO when applications expect environment variables or ordinary Secrets and you want one place to manage many secret stores.

Secrets Store CSI Driver with ASCP

Install the driver and the AWS provider (AWS publishes the provider as an EKS add-on):

helm repo add secrets-store-csi-driver \
  https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm upgrade --install csi-secrets-store \
  secrets-store-csi-driver/secrets-store-csi-driver \
  --namespace kube-system \
  --set enableSecretRotation=true \
  --set rotationPollInterval=120s
 
aws eks create-addon --cluster-name my-cluster \
  --addon-name aws-secrets-store-csi-driver-provider

Give the application’s service account a role that can read exactly its own secrets:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": "arn:aws:secretsmanager:eu-west-1:111122223333:secret:payments/*"
    }
  ]
}

Bind it with Pod Identity or IRSA. If the secret is encrypted with a customer managed KMS key, add kms:Decrypt on that key.

A SecretProviderClass declares what to fetch:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: payments-db
  namespace: payments
spec:
  provider: aws
  parameters:
    usePodIdentity: "true"
    objects: |
      - objectName: "payments/db"
        objectType: "secretsmanager"
        jmesPath:
          - path: username
            objectAlias: db-username
          - path: password
            objectAlias: db-password

# pod spec
volumes:
  - name: secrets
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: payments-db
containers:
  - name: app
    volumeMounts:
      - name: secrets
        mountPath: /mnt/secrets
        readOnly: true

The pod now sees /mnt/secrets/db-username and /mnt/secrets/db-password. If the pod’s role cannot read the secret, the pod stays in ContainerCreating with a FailedMount event that names the denied call.

Rotation

Secrets Manager can rotate a secret on a schedule with a Lambda function. Getting the new value into a running pod is a separate problem:

• Mounted files are refreshed by the driver every rotationPollInterval. The application must re-read the file — watch it, or reopen it on authentication failure.
• Environment variables never change in a running process. A rotation requires a restart; tools such as Reloader trigger a rolling restart when a synced Secret changes.
• Use the alternating-users rotation strategy for databases so the previous credential stays valid while pods catch up.

Polling has a cost: every poll is a GetSecretValue call per mounted secret per pod. With thousands of pods, lengthen the interval.

Syncing to a Kubernetes Secret

Both tools can materialise a Kubernetes Secret. With the CSI driver that is the secretObjects field, and the Secret exists only while some pod mounts the volume. Be clear about what this gives up: the value is now in etcd, readable by anyone with get secrets in the namespace. If you sync, also:

• turn on envelope encryption of Secrets with KMS (new EKS clusters encrypt all API data by default),
• restrict get/list on Secrets with RBAC.

External Secrets Operator in brief

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: payments-db
  namespace: payments
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-secrets-manager
  target:
    name: payments-db
  dataFrom:
    - extract:
        key: payments/db

The controller’s role is the security boundary here. A single ClusterSecretStore with broad read access lets any namespace request any secret; scope stores per namespace or restrict them with conditions on the store.

Secrets Manager or Parameter Store

Secrets Manager adds built-in rotation, cross-region replication and resource policies, and charges per secret per month. Parameter Store SecureString parameters are free at the standard tier and adequate for configuration that rarely changes. Both work with both tools.

Related

• Secrets management on EKS
• Sealed Secrets — the Git-native alternative
• Secrets (concepts)
• Secrets in the Security section
• Use Secrets Manager secrets in EKS