Acme Labs — Kubernetes-on-Docker Practice Lab (kind)¶
Note: this file is written for kind. If you use minikube, follow
acme-labs-minikube-learning-plan.md(stage order + a kind→minikube command translation table in its §3b); the lab content below stays valid.
Practise the Kubernetes layer of the GCP plan on your laptop first: cluster with chosen CIDRs, routes and "doors", NetworkPolicy, Istio mesh, 4-container pods with central logging, LoadBalancer ingress, and Flux ordering. Companion to acme-labs-plan.md (GCP phases P3–P9 map to labs K1–K10 here).
0. Your question: can Kubernetes create a VPC or handle subnet-to-subnet communication?¶
Short answer: no — Kubernetes sits on top of the network; it does not create it.
| Question | Answer |
|---|---|
| Can Kubernetes create a VPC / subnet / route / NAT? | No, not natively. Those are cloud (or Docker) objects. Terraform does it on GCP. |
| Can anything inside Kubernetes create them? | Only by calling the cloud API: Config Connector (GCP), Crossplane, or the Terraform/OpenTofu controller (Flux tf-controller) turn VPCs into Kubernetes CRDs — but GCP still creates the VPC. Not the learning path you want; you want hand-written Terraform. |
| Does Kubernetes know about VPC subnets? | No. It only knows nodes, a pod CIDR, and a service CIDR. Which subnet a node sits in is invisible to it. |
| How do pods in different "subnets" talk? | Kubernetes assumes a flat pod network: every pod can reach every pod. The CNI (Cilium on GKE Dataplane V2) delivers it; on GKE the pod range is a secondary range of the node subnet, so the VPC routes it natively. |
| Who controls what may talk to what? | Inside the cluster: NetworkPolicy (+ Istio). Between subnets/VPCs: VPC routes + firewall rules (cloud layer). Two layers, different owners. |
| What does Kubernetes ask the cloud for? | The cloud controller manager creates a load balancer when you make a Service type=LoadBalancer, GKE creates firewall rules/routes for the cluster, etc. |
So on Docker we simulate the layer below Kubernetes ourselves — and that is the point of this lab:
| GCP concept | Local stand-in in this lab |
|---|---|
VPC acme-main |
Docker network acme-main (10.10.0.0/24) |
Second subnet ops |
Docker network acme-ops (10.10.1.0/28) |
| Routes / Door 4 (hybrid) | A router container with ip_forward + ip route add on the nodes |
| VPC firewall | iptables rules on the router container |
| Default internet route (deleted in P8) | ip route del default inside the kind node containers |
| Doors 2 / 3 (/32 route, tag-gated default route) | Specific route / default route on selected nodes (+ a node label as the "tag") |
| Cloud NAT (SNAT, static IP) | Docker's MASQUERADE + Cilium masquerade (pod → node IP) |
| Node pool + tags | kind worker nodes + node labels |
| Pod / service secondary ranges | kind podSubnet / serviceSubnet |
| GKE Dataplane V2 | Cilium (same engine) |
| Internal load balancer | MetalLB handing out IPs from the Docker subnet |
| Cloud Logging central bucket | A Fluent Bit aggregator Deployment printing everything |
| Terraform → Flux handshake | Flux on kind (GitLab repo) |
What you will not be able to practise locally (do it on GCP)¶
Workload Identity / WIF, Private Google Access + private DNS VIP, real Cloud NAT with static IPs, Cloud Logging API, GKE private control plane + authorized networks, Artifact Registry. Everything else in the Kubernetes layer transfers 1:1.
1. Prereqs and sizing¶
- Docker (Desktop or Engine), 8 GB RAM free for Docker (cluster + Istio + Cilium). 1 control-plane + 2 workers.
- Tools:
kind,kubectl,helm,istioctl,flux,jq,dig.
# macOS
brew install kind kubectl helm istioctl fluxcd/tap/flux jq
# Linux: use each project's release binary; verify with the --version commands below
kind --version; kubectl version --client; helm version; istioctl version --remote=false; flux --version
- Docker Desktop (macOS/Windows) caveat: your host cannot reach container IPs (
10.10.x.x). Run every network test from a container (theprobecontainer in K3) or viakubectl exec. On Linux the host can reach them directly. - Linux caveat: if kind nodes crash with "too many open files":
sudo sysctl fs.inotify.max_user_watches=524288 fs.inotify.max_user_instances=512. - Work in a folder, e.g.
mkdir -p ~/acme-k8s-lab && cd ~/acme-k8s-lab.
Order: K1 → K2 → K3 → K4 → K5 → K6 → K7 → K8 → K9 → K10 → K11 (stretch). ≈ 8–12 hours total.
K1 — Docker network = VPC; cluster with explicit CIDRs¶
Goal: a kind cluster living on your subnet, with pod/service ranges you chose, no default CNI (so we can install Cilium, like GKE Dataplane V2). Nuances: pod/service CIDRs are fixed at creation; the node subnet must not overlap them; nodes are just containers with IPs on a bridge.
# the "VPC" and the "ops" subnet
docker network create --driver bridge --subnet 10.10.0.0/24 --gateway 10.10.0.1 acme-main
docker network create --driver bridge --subnet 10.10.1.0/28 --gateway 10.10.1.1 acme-ops
kind-acme.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: acme
networking:
disableDefaultCNI: true # Cilium instead of kindnet (kindnet != Dataplane V2)
podSubnet: "10.20.0.0/20" # same as the GCP gke-pods range
serviceSubnet: "10.30.0.0/24" # same as gke-svcs
nodes:
- role: control-plane
- role: worker
- role: worker
# attach the cluster to OUR docker network (experimental env var, supported by kind)
KIND_EXPERIMENTAL_DOCKER_NETWORK=acme-main kind create cluster --config kind-acme.yaml
kubectl config use-context kind-acme
kubectl get nodes # NotReady is expected: no CNI yet
helm repo add cilium https://helm.cilium.io && helm repo update
helm install cilium cilium/cilium -n kube-system \
--set ipam.mode=kubernetes \
--set hubble.relay.enabled=true --set hubble.ui.enabled=true
kubectl -n kube-system rollout status ds/cilium --timeout=300s
kubectl get nodes # Ready now
# "node pool" label (stand-in for GKE node pool / tags)
kubectl label node acme-worker pool=main
kubectl label node acme-worker2 pool=main
Hints: if Cilium pods crash-loop, your Docker kernel may be too old for eBPF — fall back to Calico (disableDefaultCNI stays, install the Calico manifest with the same pod CIDR) and note what you lose (Hubble, identity-based policy).
Verify: docker network inspect acme-main --format '{{range .Containers}}{{.Name}} {{.IPv4Address}}{{"\n"}}{{end}}' shows the 3 kind nodes in 10.10.0.0/24.
Gate G-K1: explain why a second cluster on the same Docker network with pod CIDR 10.20.0.0/20 would be a problem (overlap) — and which GCP object enforces the same rule.
K2 — Read the network you built¶
Goal: see the three address spaces (node / pod / service) and who owns each. (GCP: IPAM table in plan §2.)
kubectl get nodes -o wide # INTERNAL-IP = 10.10.0.x
kubectl get nodes -o custom-columns=NAME:.metadata.name,POD_CIDR:.spec.podCIDR,POOL:.metadata.labels.pool
kubectl get svc kubernetes -o jsonpath='{.spec.clusterIP}{"\n"}' # in 10.30.0.0/24
kubectl get pods -A -o wide # some pods show NODE IPs, others 10.20.x
docker exec acme-worker ip -br addr # eth0 = node IP on acme-main
docker exec acme-worker ip route # default via 10.10.0.1 <- the "internet gateway"
docker exec acme-worker ip route | grep 10.20 # routes/devices Cilium added for pod CIDRs
Questions to answer in your own words:
- Which pods have a node IP, and why? (hint:
hostNetwork: true) - What is
/24per node of the/20pod range? How many nodes max? (compare plan:max_pods_per_node=64→ /25 on GKE) - Which of the three ranges is routable by anything outside the cluster? (none of pods/services — this is why masquerade exists, K4.)
Gate G-K2: you can fill an IPAM table (node, pod, service, router/proxy) for this lab from memory.
K3 — Second subnet, a router, and a firewall (routes + firewall layer)¶
Goal: reproduce "subnet ↔ subnet", "route next hop" and "firewall rules" which Kubernetes itself knows nothing about. (GCP: P3 routes/firewall, Door 4.)
acme-ops 10.10.1.0/28 acme-main 10.10.0.0/24
[probe 10.10.1.3] ── 10.10.1.2 [vpc-router] 10.10.0.250 ── [acme-control-plane][acme-worker][acme-worker2]
ip_forward + iptables (route next hop + firewall)
# router: dual-homed container with IP forwarding
docker run -d --name vpc-router --cap-add NET_ADMIN --sysctl net.ipv4.ip_forward=1 \
--network acme-main --ip 10.10.0.250 nicolaka/netshoot sleep infinity
docker network connect --ip 10.10.1.2 acme-ops vpc-router
# a VM in the ops subnet (IAP-bastion stand-in)
docker run -d --name probe --cap-add NET_ADMIN --network acme-ops --ip 10.10.1.3 nicolaka/netshoot sleep infinity
# routes: "next hop = router" in both directions
docker exec probe ip route add 10.10.0.0/24 via 10.10.1.2
for n in $(kind get nodes --name acme); do docker exec $n ip route add 10.10.1.0/28 via 10.10.0.250; done
CP_IP=$(docker inspect -f '{{(index .NetworkSettings.Networks "acme-main").IPAddress}}' acme-control-plane)
docker exec probe ping -c2 $CP_IP
docker exec probe curl -sk -m5 -o /dev/null -w '%{http_code}\n' https://$CP_IP:6443/version # any HTTP code = reachable; 000 = blocked
Firewall exercise (iptables on the router = VPC firewall):
docker exec vpc-router iptables -P FORWARD DROP # "deny all" between subnets
docker exec vpc-router iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
docker exec vpc-router iptables -A FORWARD -p icmp -j ACCEPT
docker exec probe ping -c2 $CP_IP # works
docker exec probe curl -sk -m5 -o /dev/null -w '%{http_code}\n' https://$CP_IP:6443/version # 000: blocked
docker exec vpc-router iptables -A FORWARD -s 10.10.1.3 -d $CP_IP -p tcp --dport 6443 -j ACCEPT # = "authorized network"
docker exec probe curl -sk -m5 -o /dev/null -w '%{http_code}\n' https://$CP_IP:6443/version # reachable again
docker exec vpc-router iptables -L FORWARD -n -v --line-numbers
Hints:
- If
iptableserrors, tryiptables-legacy(nft/legacy mismatch on some hosts). - Routes live in the container's network namespace: they are lost when a node container restarts. Keep the
docker exec … ip route …lines inroutes.shand re-run. - Break it: remove the
ESTABLISHED,RELATEDrule → replies are dropped → "ping works one way" confusion = the exact symptom of an asymmetric firewall. Nuance revealed: Kubernetes saw nothing of this. API-server reachability from a "management subnet" is decided by route + firewall, exactly as the GKE master authorized-networks + firewall story. Gate G-K3: predict, before running, whatcurlreturns after each firewall change.
K4 — Egress: delete the default route, open the four doors¶
Goal: the plan's P8 in miniature. Cloud NAT ⇒ Docker's MASQUERADE; "tags" ⇒ node labels; "default route deleted" ⇒ ip route del default.
Preload everything the nodes will need before cutting egress (nodes can't pull images afterwards — a real lesson):
docker pull curlimages/curl && kind load docker-image curlimages/curl --name acme
docker pull nginxinc/nginx-unprivileged:alpine && kind load docker-image nginxinc/nginx-unprivileged:alpine --name acme
docker run -d --name echo --network acme-main --ip 10.10.0.201 nginx:alpine # an "internal service" that logs client IPs
Helper (run a curl pod pinned to a given node):
probe() { n=$1; shift
kubectl run p-$RANDOM --rm -i --restart=Never -q --image=curlimages/curl --image-pull-policy=Never \
--overrides="{\"spec\":{\"nodeSelector\":{\"kubernetes.io/hostname\":\"$n\"}}}" -- -sS -m 5 -o /dev/null -w '%{http_code}\n' "$@" 2>&1 | tail -1; }
EX=$(dig +short example.com | head -1); echo $EX
| Step | Command | Expected |
|---|---|---|
| 0 baseline (training wheels) | probe acme-worker https://example.com |
200 (default route → Docker gateway → host NAT) |
| 1 who do I look like? | probe acme-worker http://10.10.0.201 then docker logs echo --tail 2 |
client IP = node IP, not pod IP (SNAT). If you see a pod IP → kubectl -n kube-system exec ds/cilium -- cilium config view \| grep -i masq |
| 2 close the doors | for n in acme-worker acme-worker2; do docker exec $n ip route del default; done |
|
| 3 | probe acme-worker https://example.com |
000/timeout — likely DNS still resolves (Docker's resolver is outside the node route) but TCP has no route |
| 4 Door 2 (/32) | docker exec acme-worker ip route add $EX/32 via 10.10.0.1; probe acme-worker --resolve example.com:443:$EX https://example.com |
works for that IP only |
| 5 Door 3 (tag-gated default) | kubectl label node acme-worker2 allowed-open-nat=true; docker exec acme-worker2 ip route add default via 10.10.0.1; probe acme-worker2 https://example.com vs probe acme-worker … |
works on worker2 only |
| 6 Door 4 (hybrid via router) | probe acme-worker http://10.10.1.3 (the ops-subnet "on-prem" host; needs K3 routes) |
reaches via the router route, unaffected by the missing default |
| 7 Door 1 analogue | images arrive by kind load (side channel) — no Private-Google-Access equivalent locally |
Do this one on GCP (P3/P8) |
Hints: if Cilium agents misbehave after the route is gone ("unable to determine direct routing device"), pin devices: helm upgrade cilium cilium/cilium -n kube-system --reuse-values --set devices=eth0.
Break-it: kubectl delete pod something that needs an image not preloaded → ImagePullBackOff — the same failure you will see on GCP when Door 1/AR mirror is missing.
Optional (masquerade): helm upgrade cilium … --reuse-values --set enableIPv4Masquerade=false, restart agents, repeat step 1 → replies to an un-routable pod IP are lost. That is why GKE needs ip-masq-agent for on-prem traffic (S2).
Gate G-K4: explain why a pod inherits its node's egress ability (it is the node's route table that decides), and why per-pod egress control needs NetworkPolicy/Istio, not routes or tags.
K5 — Three domains: workloads on a flat network¶
Goal: the three namespaces, service accounts and Services; observe that Kubernetes is flat by default — everything can reach everything.
kubectl create ns observability # used in K8
for d in orders payments inventory; do
kubectl apply -f - <<EOF
apiVersion: v1
kind: Namespace
metadata: { name: dom-$d }
---
apiVersion: v1
kind: ServiceAccount
metadata: { name: $d, namespace: dom-$d }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: $d, namespace: dom-$d }
spec:
replicas: 1
selector: { matchLabels: { app: $d } }
template:
metadata: { labels: { app: $d, domain: $d } }
spec:
serviceAccountName: $d
containers:
- name: app
image: nginxinc/nginx-unprivileged:alpine # listens on 8080, has busybox wget
imagePullPolicy: IfNotPresent
ports: [{ containerPort: 8080 }]
resources: { requests: { cpu: 50m, memory: 64Mi } }
---
apiVersion: v1
kind: Service
metadata: { name: $d, namespace: dom-$d }
spec:
selector: { app: $d }
ports: [{ name: http, port: 80, targetPort: 8080 }]
EOF
done
kubectl get pods -A -l 'domain'
# helper: call <from-ns> <from-deploy> <url> -> prints the last line wget prints
call() { kubectl -n $1 exec deploy/$2 -c app -- sh -c "wget -O /dev/null -T3 $3 2>&1 | tail -n1"; }
call dom-orders orders http://payments.dom-payments/
call dom-orders orders http://inventory.dom-inventory/
call dom-payments payments http://inventory.dom-inventory/ # works too — flat network!
Nuance revealed: Service port 80 → pod 8080. Remember this for K6: NetworkPolicy ports are pod ports (8080), not Service ports.
Gate G-K5: list three reasons "it works" here is a problem for a three-domain platform.
K6 — NetworkPolicy: default-deny + explicit allows (L3/L4 guard)¶
Goal: call graph orders → payments, orders → inventory; payments ↔ inventory denied. Cilium enforces it (as on GKE Dataplane V2).
for ns in dom-orders dom-payments dom-inventory; do
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: $ns }
spec: { podSelector: {}, policyTypes: [Ingress, Egress] }
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: allow-dns, namespace: $ns }
spec:
podSelector: {}
policyTypes: [Egress]
egress:
- to:
- namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } }
ports: [{ protocol: UDP, port: 53 }, { protocol: TCP, port: 53 }]
EOF
done
# orders may call the other two domains on pod port 8080
kubectl apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: orders-egress, namespace: dom-orders }
spec:
podSelector: {}
policyTypes: [Egress]
egress:
- to:
- namespaceSelector:
matchExpressions:
- { key: kubernetes.io/metadata.name, operator: In, values: [dom-payments, dom-inventory] }
ports: [{ protocol: TCP, port: 8080 }]
EOF
for d in payments inventory; do
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: allow-from-orders, namespace: dom-$d }
spec:
podSelector: {}
policyTypes: [Ingress]
ingress:
- from: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: dom-orders } } }]
ports: [{ protocol: TCP, port: 8080 }]
EOF
done
Verify matrix (predict first):
call dom-orders orders http://payments.dom-payments/ # OK
call dom-orders orders http://inventory.dom-inventory/ # OK
call dom-payments payments http://inventory.dom-inventory/ # timeout (egress default-deny)
call dom-inventory inventory http://payments.dom-payments/ # timeout
# see the drops (agent has the hubble CLI; or install hubble and port-forward)
kubectl -n kube-system exec ds/cilium -- hubble observe --verdict DROPPED --last 10
Break-it exercises:
- Delete
allow-dnsindom-orders→ calls fail with "bad address" (DNS), not a timeout. Different symptom, same root cause class. - Change the policy port to
80→ blocked. (Service port vs pod port.) Gate G-K6: you can tell, from the error text alone, DNS-denied vs connection-denied vs not-allowed-by-server.
K7 — Istio sidecars: mTLS and identity-based authorization (L7 guard)¶
Goal: the same call graph protected by who is calling, not just where from. (GCP: P5/P7.)
istioctl install --set profile=minimal -y # istiod only (sidecar mode on purpose)
kubectl -n istio-system get pods
kubectl label ns dom-orders dom-payments dom-inventory istio-injection=enabled
for ns in dom-orders dom-payments dom-inventory; do kubectl -n $ns rollout restart deploy; done
kubectl -n dom-orders get pods # watch: pods stay 1/2 Ready ... why?
Expected break #1: default-deny egress blocks sidecar → istiod (15012), so Envoy never gets config. Prove it, then fix:
istioctl proxy-status
kubectl -n dom-orders logs deploy/orders -c istio-proxy --tail=5 # connection timeouts to istiod
for ns in dom-orders dom-payments dom-inventory; do
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: allow-istiod, namespace: $ns }
spec:
podSelector: {}
policyTypes: [Egress]
egress:
- to: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: istio-system } } }]
ports: [{ protocol: TCP, port: 15012 }]
EOF
done
kubectl -n dom-orders get pods # 2/2
istioctl proxy-status # SYNCED
call dom-orders orders http://payments.dom-payments/ # still OK
STRICT mTLS (and the "plain client" test):
kubectl apply -f - <<'EOF'
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata: { name: default, namespace: dom-payments }
spec: { mtls: { mode: STRICT } }
EOF
# temporary hole in the NetworkPolicy so the plain client can reach the port at all
kubectl apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: tmp-allow-from-default, namespace: dom-payments }
spec:
podSelector: {}
policyTypes: [Ingress]
ingress:
- from: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: default } } }]
ports: [{ protocol: TCP, port: 8080 }]
EOF
kubectl run plain --rm -i --restart=Never -q --image=curlimages/curl --image-pull-policy=Never -- -sS -m4 http://payments.dom-payments/ # reset / empty reply = STRICT mTLS
kubectl -n dom-payments delete netpol tmp-allow-from-default
call dom-orders orders http://payments.dom-payments/ # meshed client: OK
Identity-based AuthorizationPolicy — an "intruder" in the same namespace as orders (the network layer cannot tell them apart; the identity can):
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata: { name: intruder, namespace: dom-orders }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: intruder, namespace: dom-orders }
spec:
replicas: 1
selector: { matchLabels: { app: intruder } }
template:
metadata: { labels: { app: intruder } }
spec:
serviceAccountName: intruder
containers:
- { name: app, image: nginxinc/nginx-unprivileged:alpine, imagePullPolicy: IfNotPresent }
EOF
kubectl apply -f - <<'EOF'
apiVersion: security.istio.io/v1 # older Istio: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata: { name: only-orders, namespace: dom-payments }
spec:
action: ALLOW
rules:
- from: [{ source: { principals: ["cluster.local/ns/dom-orders/sa/orders"] } }]
EOF
call dom-orders orders http://payments.dom-payments/ # OK
call dom-orders intruder http://payments.dom-payments/ # "403 Forbidden" (RBAC: access denied) — network allowed, identity not
Hints: istioctl analyze -A; istioctl x describe pod <pod> -n dom-orders; kubectl -n dom-payments logs deploy/payments -c istio-proxy.
Nuance revealed: two guards, two failure signatures: timeout (NetworkPolicy) vs 403 (AuthorizationPolicy) vs reset (STRICT mTLS vs plaintext).
Gate G-K7: for each of the three failures, name the layer without running a command.
K8 — Four-container pod and central logging¶
Goal: app + istio-proxy (injected) + fluent-bit + otel-collector in one pod; everything lands in one central place (stand-in for Cloud Logging bucket + sink).
8a. Central aggregator (the "log bucket")¶
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: { name: aggregator-config, namespace: observability }
data:
fluent-bit.conf: |
[SERVICE]
Flush 1
[INPUT]
Name forward
Listen 0.0.0.0
Port 24224
[OUTPUT]
Name stdout
Match *
Format json_lines
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: log-aggregator, namespace: observability }
spec:
replicas: 1
selector: { matchLabels: { app: log-aggregator } }
template:
metadata: { labels: { app: log-aggregator } }
spec:
containers:
- name: fluent-bit
image: fluent/fluent-bit:3.2
ports: [{ containerPort: 24224 }]
volumeMounts: [{ name: cfg, mountPath: /fluent-bit/etc/fluent-bit.conf, subPath: fluent-bit.conf }]
volumes: [{ name: cfg, configMap: { name: aggregator-config } }]
---
apiVersion: v1
kind: Service
metadata: { name: log-aggregator, namespace: observability }
spec:
selector: { app: log-aggregator }
ports: [{ name: tcp-forward, port: 24224, targetPort: 24224 }]
EOF
8b. The 4-container pod in dom-orders¶
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: { name: logdemo-fluentbit, namespace: dom-orders }
data:
fluent-bit.conf: |
[SERVICE]
Flush 1
Parsers_File parsers.conf
[INPUT]
Name tail
Path /var/log/app/*.log
Parser json
Tag app.${DOMAIN}
[FILTER]
Name record_modifier
Match *
Record domain ${DOMAIN}
[OUTPUT]
Name forward
Match *
Host log-aggregator.observability.svc.cluster.local
Port 24224
---
apiVersion: v1
kind: ConfigMap
metadata: { name: logdemo-otel, namespace: dom-orders }
data:
config.yaml: |
receivers:
otlp:
protocols:
http: { endpoint: 127.0.0.1:4318 }
exporters:
debug: { verbosity: basic }
service:
pipelines:
traces: { receivers: [otlp], exporters: [debug] }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: logdemo, namespace: dom-orders }
spec:
replicas: 1
selector: { matchLabels: { app: logdemo } }
template:
metadata: { labels: { app: logdemo } }
spec:
serviceAccountName: orders
volumes:
- { name: applogs, emptyDir: {} }
- { name: fb-cfg, configMap: { name: logdemo-fluentbit } }
- { name: otel-cfg, configMap: { name: logdemo-otel } }
containers:
- name: app # writes JSON to stdout AND a file, emits one OTLP span
image: curlimages/curl
imagePullPolicy: Never
env: [{ name: DOMAIN, value: orders }]
command: ["/bin/sh", "-c"]
args:
- |
while true; do
echo "{\"severity\":\"INFO\",\"message\":\"tick\",\"domain\":\"${DOMAIN}\"}" | tee -a /var/log/app/app.log
curl -s -m2 -H 'Content-Type: application/json' http://127.0.0.1:4318/v1/traces -d '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"orders"}}]},"scopeSpans":[{"spans":[{"traceId":"5B8EFFF798038103D269B633813FC60C","spanId":"EEE19B7EC3C1B174","name":"tick","startTimeUnixNano":"1700000000000000000","endTimeUnixNano":"1700000001000000000","kind":1}]}]}]}' >/dev/null
sleep 5
done
volumeMounts: [{ name: applogs, mountPath: /var/log/app }]
- name: fluent-bit # file-tail sidecar
image: fluent/fluent-bit:3.2
env: [{ name: DOMAIN, value: orders }]
volumeMounts:
- { name: applogs, mountPath: /var/log/app, readOnly: true }
- { name: fb-cfg, mountPath: /fluent-bit/etc/fluent-bit.conf, subPath: fluent-bit.conf } # subPath keeps parsers.conf
- name: otel-collector # OTLP sidecar
image: otel/opentelemetry-collector:latest # pin a version once it works
args: ["--config=/conf/config.yaml"]
volumeMounts: [{ name: otel-cfg, mountPath: /conf }]
EOF
kubectl -n dom-orders get pod -l app=logdemo -o jsonpath='{.items[0].spec.containers[*].name}{"\n"}' # app fluent-bit otel-collector istio-proxy
Expected break #2: the fluent-bit sidecar cannot reach the aggregator (default-deny egress from dom-orders). Fix by allowing it, then watch the central log:
kubectl apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: allow-to-observability, namespace: dom-orders }
spec:
podSelector: {}
policyTypes: [Egress]
egress:
- to: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: observability } } }]
ports: [{ protocol: TCP, port: 24224 }]
EOF
kubectl -n dom-orders logs deploy/logdemo -c otel-collector --tail=5 # "Traces" lines = OTLP path alive
kubectl -n observability logs deploy/log-aggregator --tail=5 # JSON lines with "domain":"orders" = file-tail path alive
kubectl -n dom-orders logs deploy/logdemo -c app --tail=3 # stdout path (what the GKE node agent would ship)
Exercises:
- Turn this pod spec into the Helm chart
acme-microservice(plan P7a):helm create acme-microservice, delete the defaults, keepvalues.yamltogglessidecars.otel.enabled,sidecars.fluentbit.enabled,domain,image.*; verify withhelm lintandhelm template . --set domain=payments | kubectl apply --dry-run=client -f -. - Add the same pod to
dom-paymentswithdomain=paymentsand filter by thedomainfield at the aggregator. - Real images: build your own Go/Python/Node service and
docker build→kind load docker-image(the local stand-in for Artifact Registry). Gate G-K8: explain the three log paths (stdout, file-tail, OTLP) and which of them needs a NetworkPolicy hole.
K9 — LoadBalancer ingress: MetalLB (internal LB analogue)¶
Goal: the P9 "internal load balancer" and its three classic breaks, using the ops subnet as the client side.
# check the current release at github.com/metallb/metallb before pinning
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.8/config/manifests/metallb-native.yaml
kubectl -n metallb-system rollout status deploy/controller --timeout=180s
kubectl apply -f - <<'EOF'
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata: { name: acme-pool, namespace: metallb-system }
spec: { addresses: ["10.10.0.240-10.10.0.249"] }
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata: { name: acme-l2, namespace: metallb-system }
EOF
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Service
metadata: { name: payments-lb, namespace: dom-payments }
spec:
type: LoadBalancer
externalTrafficPolicy: Local # keep the client's source IP
selector: { app: payments }
ports: [{ name: http, port: 80, targetPort: 8080 }]
EOF
kubectl -n dom-payments get svc payments-lb # EXTERNAL-IP from 10.10.0.240-249
LB=$(kubectl -n dom-payments get svc payments-lb -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
docker exec probe curl -s -m4 -o /dev/null -w '%{http_code}\n' http://$LB/
Expected breaks (diagnose each by symptom, then fix):
| # | Symptom from probe |
Layer | Fix |
|---|---|---|---|
| 1 | timeout | router firewall (K3 FORWARD DROP) |
iptables -A FORWARD -s 10.10.1.3 -d $LB -p tcp --dport 80 -j ACCEPT (+ established rule) |
| 2 | timeout (router now open) | NetworkPolicy ingress deny on dom-payments |
add ingress from ipBlock 10.10.1.0/28 on pod port 8080 |
| 3 | connection reset | STRICT mTLS vs plaintext client (an LB speaks plaintext) | workload-level PeerAuthentication with portLevelMtls: { 8080: { mode: PERMISSIVE } } (needs selector.matchLabels app: payments) |
| 4 | 403 |
AuthorizationPolicy — plaintext request has no principal | add a second ALLOW rule from: source: ipBlocks: ["10.10.1.0/28"] (verify the source IP Envoy sees) |
Nuance revealed: every layer you built in K3–K7 appears again in the ingress path — firewall, NetworkPolicy, mTLS mode, authorization — which is why P9 on GCP also has 3–4 breaks. Gate G-K9: from the symptom alone, name the failing layer for each row.
K10 — Flux: let Git own the cluster (ordering lab)¶
Goal: replace every manual kubectl apply of K5–K9 with a Flux repo whose ordering is explicit. (GCP: P5.)
Note: the CNI (Cilium) is installed outside Flux — Flux pods themselves need a network. On GKE the equivalent is Dataplane V2 being a cluster setting.
export GITLAB_TOKEN=<PAT with api scope>
flux check --pre
flux bootstrap gitlab --hostname=gitlab.com \
--owner=acme-labs-<handle>/domain-shared/configurations --repository=flux-releases-kind \
--branch=main --path=clusters/kind --token-auth # add --personal if the owner is a user, not a group
flux get kustomizations -A
Repo layout:
clusters/kind/
flux-system/ (generated)
10-infra-controllers.yaml
20-infra-configs.yaml
30-apps.yaml
infrastructure/controllers/ sources.yaml istio-base.yaml istiod.yaml metallb.yaml
infrastructure/configs/ namespaces.yaml netpol.yaml mesh-policy.yaml metallb-pool.yaml
apps/kind/ observability.yaml orders.yaml payments.yaml inventory.yaml
clusters/kind/10-infra-controllers.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata: { name: infra-controllers, namespace: flux-system }
spec:
interval: 10m
path: ./infrastructure/controllers
prune: true
wait: true
timeout: 10m
sourceRef: { kind: GitRepository, name: flux-system }
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata: { name: infra-configs, namespace: flux-system }
spec:
interval: 10m
dependsOn: [{ name: infra-controllers }]
path: ./infrastructure/configs
prune: true
wait: true
sourceRef: { kind: GitRepository, name: flux-system }
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata: { name: apps, namespace: flux-system }
spec:
interval: 10m
dependsOn: [{ name: infra-configs }]
path: ./apps/kind
prune: true
sourceRef: { kind: GitRepository, name: flux-system }
infrastructure/controllers/sources.yaml + Istio releases (MetalLB analogous, chart repo https://metallb.github.io/metallb, chart metallb)
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata: { name: istio, namespace: flux-system }
spec: { interval: 1h, url: https://istio-release.storage.googleapis.com/charts }
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata: { name: istio-base, namespace: flux-system }
spec:
interval: 10m
targetNamespace: istio-system
install: { createNamespace: true }
chart: { spec: { chart: base, sourceRef: { kind: HelmRepository, name: istio, namespace: flux-system } } }
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata: { name: istiod, namespace: flux-system }
spec:
interval: 10m
targetNamespace: istio-system
dependsOn: [{ name: istio-base }]
chart: { spec: { chart: istiod, sourceRef: { kind: HelmRepository, name: istio, namespace: flux-system } } }
values:
meshConfig: { accessLogFile: /dev/stdout, accessLogEncoding: JSON }
Migration exercise (do it in this order):
- Commit the K5 namespaces (with
istio-injection: enabledlabel), K6/K7 policies and K9 pool intoconfigs/, the Deployments intoapps/kind/; commit the controllers intocontrollers/. kubectl deletethe manually-applied objects (or recreate the cluster from K1) and watch:flux get ks -A -w,flux tree ks apps,flux events.- After each change:
flux reconcile ks apps --with-source.
Break-it exercises (the point of this lab):
- Remove
dependsOnfrominfra-configs→ the pool fails with "no matches for kind IPAddressPool" (CRDs not yet installed). Put it back. - Create the namespaces after the Deployments (move the label out) → pods start without a sidecar; restart them to fix. This is the plan's G5.
- Remove
wait: truefrom controllers → dependants start before istiod is ready. Gate G-K10: you can state, for each object, why it is in controllers vs configs vs apps.
K11 — Stretch (local versions of the GCP stretch menu)¶
| Idea | How locally | GCP twin |
|---|---|---|
| Two clusters / two VPCs | second kind cluster on acme-ops network with different pod/service CIDRs, router between them |
S2 hybrid, overlapping-CIDR pain |
| Per-domain node pool | label workers pool=orders etc., nodeSelector + taints per domain |
S3 |
| Egress with a fixed IP | Cilium Egress Gateway (check current docs for status/requirements) | Cloud NAT static IPs |
| Registry mirror | registry:2 container on acme-main + containerd mirror config (see kind "local registry" docs) |
Artifact Registry + Door 1 |
| Mesh REGISTRY_ONLY | meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY + ServiceEntry |
S5 |
| Observability | replace the aggregator with Loki + Grafana | Log Analytics |
Gates summary (predict first, then verify)¶
| Gate | You can… |
|---|---|
| G-K1 | explain why CIDRs must not overlap (node / pod / service / peer) |
| G-K2 | fill the IPAM table from memory |
| G-K3 | predict each curl result across firewall changes |
| G-K4 | explain why pods inherit their node's egress and why tags can't do per-pod control |
| G-K5 | list three risks of a flat network |
| G-K6 | tell DNS-denied / connection-denied / server-denied apart from text |
| G-K7 | name the layer for timeout / reset / 403 |
| G-K8 | explain stdout vs file-tail vs OTLP |
| G-K9 | name the failing layer for each ingress symptom |
| G-K10 | justify controllers → configs → apps order |
When all ten pass, repeat on GCP (plan P3–P9). Expect only the cloud-specific pieces (WIF, Private Google Access, Cloud NAT, Cloud Logging, authorised networks) to be new.
Troubleshooting¶
| Symptom | Likely cause |
|---|---|
Nodes NotReady after create |
CNI not installed yet (K1) |
| Cilium pods CrashLoop | old kernel / Docker VM without eBPF → Calico fallback |
Pods ImagePullBackOff after K4 |
default route removed; image not kind load-ed |
| Routes vanished | node container restarted → re-run your routes.sh |
Can't reach 10.10.x.x from host |
Docker Desktop VM → test from probe/kubectl exec |
| Sidecar missing | namespace label added after pods started → rollout restart |
| Pods 1/2 Ready after injection | egress to istiod:15012 blocked (K7 break #1) |
flux bootstrap fails |
token scope, group path, or --personal mismatch |