cloudnative wiki

Custom Networking

1 min read

Custom Networking with VPC CNI

Overview

Custom networking allows pods to use secondary CIDR blocks instead of the primary VPC CIDR, useful for IP address conservation.

Use Cases

  • Large-scale deployments requiring more IPs
  • Isolating pod traffic to specific CIDR
  • IP address space reuse across clusters

Configuration

Enable Custom Networking

# Set CNI configuration
kubectl set env daemonset/aws-node -n kube-system \
  AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true

Create ENIConfig for Custom Subnet

apiVersion: crd.k8s.aws/v1alpha1
kind: ENIConfig
metadata:
  name: my-custom-subnet
spec:
  subnet: subnet-0123456789abcdef0
  securityGroups:
    - sg-0123456789abcdef0

Assign ENIConfig to Nodes

apiVersion: v1
kind: Node
metadata:
  labels:
    k8s.amazonaws.com/eniConfig: my-custom-subnet

Node Group with ENIConfig

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: my-cluster
  region: us-west-2
managedNodeGroups:
  - name: custom-networking
    instanceType: t3.medium
    labels:
      k8s.amazonaws.com/eniConfig: my-custom-subnet
    annotations:
      k8s.amazonaws.com/eniConfig: my-custom-subnet

Limitations

  • Requires secondary CIDR block attached to VPC
  • Custom subnet must be in same AZ as nodes
  • Security groups work differently with custom networking

References

Graph View

Backlinks