Secrets Management on EKS
A Kubernetes Secret is a delivery mechanism, not a vault. Its values are base64-encoded, stored in etcd, and readable by anyone with get on Secrets in the namespace — and by anyone who can create a pod there. Secrets management on EKS is about deciding where the source of truth lives and how values reach pods with the least exposure.
The fundamentals are in Secrets (concepts) and secret encryption.
Notes in this section
- AWS Secrets Manager — the Secrets Store CSI Driver with the AWS provider, External Secrets Operator, rotation, and what syncing to a Kubernetes Secret gives up.
- Sealed Secrets — encrypted secrets in Git, decrypted by a controller in the cluster; scopes, key rotation and backup.
What EKS does by default
- Encryption at rest. EKS encrypts etcd volumes, and newer clusters apply envelope encryption to all Kubernetes API data by default. You can supply your own KMS key for Secrets, which adds key-policy control and an audit trail of decryptions in CloudTrail.
- Encryption in transit. All API traffic is TLS.
Neither protects against someone who is authorised to read the Secret through the API. That is what RBAC and the choices below are for.
The approaches
| Approach | Source of truth | Reaches the pod as | Stored in etcd | Rotation |
|---|---|---|---|---|
| Plain Kubernetes Secrets | The cluster | Env vars or files | Yes | Manual |
| Secrets Store CSI Driver + ASCP | Secrets Manager / SSM | Files on a tmpfs volume | No (optional) | Built in |
| External Secrets Operator | Secrets Manager / SSM | A synced Kubernetes Secret | Yes | Built in |
| Sealed Secrets | Git (encrypted) | A Kubernetes Secret | Yes | Manual |
| SOPS with KMS | Git (encrypted) | A Kubernetes Secret | Yes | Manual |
| No secret at all | IAM | Temporary credentials | No | Automatic |
The last row is the most important. Most “secrets” on EKS are AWS credentials, and those should not exist: give the pod an IAM role through Pod Identity or IRSA and let the SDK fetch short-lived credentials. The same goes for databases that support IAM authentication. The best-managed secret is the one you eliminated.
Choosing
| Situation | Recommendation |
|---|---|
| The pod needs to call AWS APIs | Pod Identity — no secret |
| Compliance requires that secrets never rest in etcd | CSI driver with mounted files, no sync |
| Applications expect environment variables or ordinary Secrets | External Secrets Operator |
| Automatic rotation and a central audit trail matter | Secrets Manager with either tool |
| Everything must live in Git; no dependency on a cloud API | Sealed Secrets |
| Secrets in Git, with AWS managing the keys | SOPS with KMS (native in Flux) |
| Fargate-only workloads | External Secrets Operator (the CSI driver needs a DaemonSet) |
Mixing is normal: Pod Identity for AWS access, Secrets Manager for third-party API keys and database passwords, and nothing sensitive in Git in plain text.
Practices that apply whichever tool you choose
- Least privilege on both sides. Narrow IAM permissions to specific secret paths, and narrow Kubernetes RBAC on
secrets—listandwatchreveal values just asgetdoes. - Prefer mounted files to environment variables. Environment variables leak into crash dumps, debug output and child processes, and never update in a running process.
- Rotate, and make applications tolerate it: re-read the file, or restart on change.
- Separate by namespace and by environment. One service account, one role, one set of secrets.
- Keep secrets out of images, manifests, logs and CI output — scan for them: secrets detection.
- Audit access. CloudTrail for Secrets Manager and KMS; Kubernetes audit logs for Secret reads.
- Plan recovery. Know how secrets are restored after a cluster loss — especially the sealing keys for Sealed Secrets.