cloudnative wiki

Lab CKA-11 — Service & DNS Troubleshooting

8 min read

Lab CKA-11 — Service & DNS Troubleshooting

Troubleshooting is 30% of the CKA, and “troubleshoot services and networking” is one of its named objectives. This lab is built differently from the guided ones: a script creates four applications that are all broken, plus one cluster-wide fault later, and you are not told what is wrong. Work through them with the method below. The answers are in collapsed blocks; open one only after you have a diagnosis you would defend.

Concepts: Service Unreachable, DNS Resolution, NetworkPolicy and EndpointSlices.

Tested 2026-10-13 on kind v0.33.0 with Kubernetes v1.37.0.

1. Prerequisites & Starting State

  • Working k8s-lab cluster from Lab CKAD-01. No other lab is required. Lab CKAD-05 introduces the mechanics this lab tests.

Save and run the setup. Do not study it; the point is to find the faults from the cluster.

cat << 'SETUP' > net-faults.sh
kubectl create namespace net-lab
for A in alpha bravo delta; do
  kubectl -n net-lab create deployment $A --image=ghcr.io/stefanprodan/podinfo:6.7.1 --port=9898
  kubectl -n net-lab expose deployment $A --port=80 --target-port=9898
done
kubectl -n net-lab create deployment charlie --image=ghcr.io/stefanprodan/podinfo:6.7.1 --port=9898 --dry-run=client -o yaml \
  | kubectl patch --local -f - --type json -o yaml -p '[{"op":"add","path":"/spec/template/spec/containers/0/readinessProbe","value":{"httpGet":{"path":"/ready","port":9898},"periodSeconds":3}}]' \
  | kubectl apply -f -
kubectl -n net-lab expose deployment charlie --port=80 --target-port=9898
kubectl -n net-lab patch service alpha -p '{"spec":{"selector":{"app":"alhpa"}}}'
kubectl -n net-lab patch service bravo --type json -p '[{"op":"replace","path":"/spec/ports/0/targetPort","value":8080}]'
kubectl -n net-lab apply -f - << 'POL'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: delta-lockdown
spec:
  podSelector:
    matchLabels:
      app: delta
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              role: gateway
POL
kubectl -n net-lab run client --image=curlimages/curl:8.10.1 --restart=Never --command -- sleep 3600
kubectl -n net-lab wait --for=condition=Ready pod/client --timeout=90s
SETUP
 
sh net-faults.sh
sleep 15

2. The Symptom

From the client Pod, none of the four Services answers:

for A in alpha bravo charlie delta; do
  printf '%-8s' $A
  kubectl -n net-lab exec client -- curl -s -m 4 -o /dev/null -w '%{http_code} (curl exit %{exitcode})\n' "http://$A/"
done
alpha   000 (curl exit 7)
bravo   000 (curl exit 7)
charlie 000 (curl exit 7)
delta   000 (curl exit 28)

The curl exit code is the first piece of evidence, before any kubectl command:

ExitMeaningPoints towards
6Could not resolve the nameDNS, or a wrong name or namespace
7Connection refusedNo endpoints, or nothing listening on the target port
28Timed outPackets dropped: a NetworkPolicy, or a host that is not there
0 with a 5xx codeThe application answeredThe application itself

Your task: make all four return 200 from client, changing as little as possible. Do not delete and recreate the applications. Suggested time: 20 minutes for the four.

3. The Method

Work from the client towards the Pod. Each step either clears a layer or names the fault.

flowchart TD
    A["1 · Does the name resolve?<br/>nslookup NAME"] -->|no| DNS["DNS: CoreDNS, the name, the namespace"]
    A -->|yes| B["2 · Does the Service have endpoints?<br/>kubectl get endpointslices"]
    B -->|none| C["3 · Do Pods match the selector?<br/>kubectl get pods -l SELECTOR"]
    C -->|no Pods| SEL["Selector or labels are wrong"]
    C -->|Pods exist, not Ready| RDY["Readiness: describe the Pod"]
    B -->|present| D["4 · Is the endpoint port one the container listens on?"]
    D -->|no| TP["targetPort is wrong"]
    D -->|yes| E["5 · Does the Pod answer when called directly by IP?"]
    E -->|yes, but not via the Service| KP["kube-proxy on the node"]
    E -->|no, times out| NP["6 · NetworkPolicy:<br/>kubectl get networkpolicy"]
    E -->|no, refused| APP["The application is not listening"]

The commands, with SVC standing for the Service:

kubectl -n net-lab exec client -- nslookup SVC.net-lab.svc.cluster.local
kubectl -n net-lab get service SVC -o wide
kubectl -n net-lab get endpointslices -l kubernetes.io/service-name=SVC
kubectl -n net-lab get pods -l <the Service's selector> -o wide --show-labels
kubectl -n net-lab get pod <pod> -o jsonpath='{.spec.containers[*].ports}'
kubectl -n net-lab exec client -- curl -s -m 4 -o /dev/null -w '%{http_code}\n' http://<pod IP>:<container port>/
kubectl -n net-lab get networkpolicy

Two overview commands show most of the picture at once. Run them and try to place each application on the flowchart before going further:

kubectl -n net-lab get pods
kubectl -n net-lab get endpointslices

4. Answers

Verify all four before continuing:

for A in alpha bravo charlie delta; do
  printf '%-8s' $A
  kubectl -n net-lab exec client -- curl -s -m 4 -o /dev/null -w '%{http_code}\n' "http://$A/"
done

5. A Fifth Fault, Cluster-Wide

With all four working, break something else. Run this, then diagnose before reading on:

kubectl -n kube-system scale deployment coredns --replicas=0
sleep 8
kubectl -n net-lab exec client -- curl -s -m 6 -o /dev/null -w '%{http_code} (curl exit %{exitcode})\n' http://delta/
000 (curl exit 6)

Exit 6: the name did not resolve. Every application in every namespace is affected at once, which itself is a clue that the fault is in something shared.

6. Cleanup & Exit State

kubectl delete namespace net-lab
kubectl -n kube-system get deployment coredns     # READY 2/2
rm -f net-faults.sh

Do not leave CoreDNS at zero replicas; every later lab depends on it.

What to Take Away

Client seesEndpointsMost likely cause
Cannot resolveNot relevantDNS, or the name is wrong
RefusedNoneSelector or labels
RefusedPresent, port looks wrongtargetPort
RefusedPresent, ready=falseReadiness probe
TimeoutPresent and readyNetworkPolicy
Works by IP, not by namePresent and readyDNS

Refused-versus-timeout is how this cluster’s kube-proxy and CNI behave. Other dataplanes can turn a refusal into a timeout, so use the client error as a first hint and the EndpointSlice as the evidence.

To repeat the lab as real practice, run the setup again after a few days and time yourself without opening the answers.

Further reading

Next Lab

Proceed to Lab CKA-12 — Gateway API, or return to the CKA Track.

Graph View

Backlinks