cloudnative wiki

Flux on EKS

5 min read

Flux on EKS

Flux is a set of Kubernetes controllers that keep a cluster in sync with sources such as Git repositories, Helm repositories and OCI artifacts. It is a CNCF graduated project and the main alternative to Argo CD. Where Argo CD is an application with a UI, Flux is a toolkit of small controllers driven entirely by custom resources. The underlying ideas are in GitOps basics.

The controllers

ControllerCustom resourcesJob
source-controllerGitRepository, HelmRepository, OCIRepository, BucketFetches sources and exposes them as versioned artifacts
kustomize-controllerKustomizationBuilds and applies manifests, prunes, health-checks
helm-controllerHelmReleaseInstalls and upgrades Helm charts, rolls back on failure
notification-controllerProvider, Alert, ReceiverSends events out; receives webhooks to trigger syncs
image-reflector / image-automationImageRepository, ImagePolicy, ImageUpdateAutomationWatches registries and commits new tags to Git

Everything is reconciled on an interval. A Kustomization can depend on another (dependsOn), which is how ordering is expressed: CRDs, then controllers, then applications.

Bootstrap

flux bootstrap github \
  --owner=example \
  --repository=fleet \
  --branch=main \
  --path=clusters/prod-eu-west-1 \
  --personal=false

Bootstrap commits Flux’s own manifests to the repository, installs them in the cluster, and points Flux at that path. From then on Flux manages itself: upgrading Flux is a commit. A typical repository layout:

fleet/
├── clusters/
│   ├── prod-eu-west-1/      # entry point: Kustomizations for this cluster
│   └── staging-eu-west-1/
├── infrastructure/          # controllers: ingress, cert-manager, CSI drivers
│   ├── base/
│   └── overlays/{prod,staging}/
└── apps/
    ├── base/
    └── overlays/{prod,staging}/

Flux is pull-based and per-cluster: every cluster runs its own Flux and pulls from Git. There is no central hub holding credentials for the fleet, which is a real security advantage and means a new cluster only needs bootstrap to converge.

Authenticating to AWS services with IAM

Give Flux’s service accounts a role through IRSA or Pod Identity instead of storing credentials.

ECR (images and OCI Helm charts). Set provider: aws and Flux exchanges the pod’s role for an ECR token automatically — no 12-hour token refresh job:

apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 5m
  url: oci://111122223333.dkr.ecr.eu-west-1.amazonaws.com/charts/podinfo
  provider: aws
  ref:
    semver: ">=6.0.0"

The source-controller role needs ecr:GetAuthorizationToken plus pull permissions on the repositories.

S3. A Bucket source with provider: aws reads manifests from S3, useful in environments where Git is not reachable from the cluster.

OCI artifacts instead of Git. CI can package rendered manifests with flux push artifact to ECR and clusters pull from there. This removes the cluster’s dependency on the Git provider and lets you sign and verify what is deployed.

Secrets: SOPS with KMS

Flux decrypts SOPS-encrypted files natively, and SOPS can use an AWS KMS key.

# .sops.yaml in the repository
creation_rules:
  - path_regex: .*/secrets/.*\.yaml
    encrypted_regex: ^(data|stringData)$
    kms: arn:aws:kms:eu-west-1:111122223333:key/1234abcd-...
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 10m
  path: ./apps/overlays/prod
  prune: true
  sourceRef: { kind: GitRepository, name: flux-system }
  decryption:
    provider: sops

Developers encrypt with sops -e; the kustomize-controller’s role needs kms:Decrypt on the key. Secrets live in Git encrypted, access is governed by the key policy, and every decryption is logged in CloudTrail. Alternatives are compared in secrets management on EKS.

Helm releases

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: podinfo
  namespace: apps
spec:
  interval: 10m
  chartRef: { kind: OCIRepository, name: podinfo, namespace: flux-system }
  install: { remediation: { retries: 3 } }
  upgrade: { remediation: { retries: 3, remediateLastFailure: true } }
  driftDetection: { mode: enabled }
  values:
    replicaCount: 3

Unlike Argo CD, which renders charts to plain manifests, Flux uses the Helm SDK: releases are real Helm releases, helm list shows them, and hooks and rollbacks behave as Helm users expect.

Image automation

The image controllers scan ECR for new tags, select one by policy (semver range, numeric order, regex), and commit the updated tag back to Git. Git stays the source of truth, and a deployment to staging becomes “CI pushed an image”. Use it for lower environments; promote to production through a pull request.

Flux or Argo CD

FluxArgo CD
TopologyAgent in every cluster, pull onlyUsually a central hub pushing to spokes
UINone built in (third-party UIs exist)Rich web UI with diff and topology views
HelmNative Helm releasesRenders templates, applies as manifests
SecretsSOPS built inPlugins or external operators
Multi-tenancyKubernetes RBAC and service-account impersonationProjects and its own RBAC
Best forPlatform teams automating fleets as codeTeams that want visibility and click-to-sync

Both are sound. The decision is mostly about whether people want a UI and whether a central control plane is acceptable.

Operating notes

  • flux get all -A and flux events are the first diagnostic commands; flux reconcile kustomization <name> --with-source forces a sync.
  • Suspend, do not delete, when intervening by hand: flux suspend kustomization apps. Otherwise Flux reverts your change at the next interval.
  • Set spec.wait: true and health checks so a Kustomization is only Ready when its workloads are.
  • Use spec.serviceAccountName on tenant Kustomizations so they apply with the tenant’s permissions, not Flux’s cluster-admin.
  • Send alerts to chat and commit statuses back to the Git provider through the notification-controller.

Across the wiki

Graph View

Backlinks