Skip to content

Lab 01 · Docker Underlay: VPC Subnets, Next-Hop Routing & Stateful Firewalls

Core Lesson: Kubernetes sits on top of the network; it cannot create VPCs, subnets, route tables, or cloud firewalls. Before starting Minikube, you will build the underlay network yourself using two Docker bridge subnets, a dual-homed Linux router container (vpc-router), and iptables connection tracking. GCP Twin: Phase P3 (google_compute_network, google_compute_subnetwork, google_compute_route, google_compute_firewall, IAP bastion VM).

RFC 1918 — Address Allocation for Private Internets

Step 0 — The Starting Point: Repos & Folders

One VPC acme-main (np) — both subnets are private; there is no public subnet because the default internet route is deleted at birth (that's Lab 04):

  • subnet gke-main 10.10.0.0/24 — the cluster subnet (nodes; pod 10.20.0.0/20 + svc 10.30.0.0/24 are secondary ranges inside it, not separate subnets)
  • subnet ops 10.10.1.0/28 — the bastion/probe subnet; admin traffic enters here and must route + pass firewall to reach the cluster
KubernetesConcepts/                 ← this repo (lab docs + runnable manifests)
├─ apps/orders  apps/payments  apps/inventory     # the 3 domain microservices
├─ apps/data/                                     # dom-data tier: Postgres + secrets + seed Job
├─ charts/acme-microservice/                      # vendored Tier-2 Helm chart
└─ src/1-pods … src/12-troubleshooting-scenarios  # concept manifest library

~/acme-k8s-lab/                     ← your scratch dir (nothing here is committed)
├─ charts/acme-microservice/        # the chart YOU author in Lab 03
├─ gitops/                          # clusters/local + infrastructure/{controllers,configs,data} + apps/local
└─ scripts/                         # cluster-up.sh, cluster-down.sh, verify-all.sh

gitlab.com/acme-labs-<handle>/      ← the GCP twin: one GitLab PROJECT per box
├─ platform/          bootstrap · network · gke-platform · observability · terraform-modules · ci-templates
├─ domain-shared/     shared-services (Cloud SQL) · configurations/helm-charts · configurations/flux-releases
└─ applications/      orders/{orders-svc,orders-infra} · payments/{…} · inventory/{…}

environments/
├─ np   = my-devops-journey-502420  → cluster acme-np, pipeline-np SA, flux clusters/np   ← labs build this
└─ prod = acme-labs-p-<suffix>      → state+SAs only, pipeline-p (protected main) / pipeline-p-ro, flux clusters/p

1. Architecture Decision: Where Does Underlay End and Kubernetes Begin?

Question Architect's Answer
Why build two Docker networks (acme-main + acme-ops)? A single subnet hides routing bugs because every host is L2-adjacent (ARP). Separating the bastion (acme-ops 10.10.1.0/28) from the cluster (acme-main 10.10.0.0/24) forces every admin and ingress packet through a Layer 3 next hop and a stateful firewall—just like an enterprise Bastion/VPN subnet talking to GKE.
Why do we need conntrack ESTABLISHED,RELATED on the firewall? GCP VPC Firewalls are stateful: if you allow probe -> API Server (6443), the return packet (6443 -> probe) is automatically allowed. Raw iptables with FORWARD DROP is stateless until you explicitly add a conntrack rule. Forgetting this creates the classic "one-way ping / asymmetric drop" bug.
Why pre-deploy a local registry (10.10.0.200) and echo server (10.10.0.201)? In Lab 04, we will delete the default internet route (0.0.0.0/0). Having 10.10.0.200:5000 on acme-main gives us a local twin of GCP Artifact Registry over Private Google Access (Door 1), and echo (10.10.0.201) logs client source IPs to prove Pod SNAT/masquerade.

2. Visual Blueprint: The Underlay Topology

Underlay topology — two Docker subnets, router, stateful firewall


3. Step-by-Step Hands-On Build

Commands live in Taskfile.yml. Each step below names its task — run task --list to see the whole build order. Inline commands that remain are verification or break/drill commands — the evidence you're meant to read, not boilerplate.

Step 1: Create the Two Subnets (acme-main & acme-ops)

task lab01:up   # creates BOTH subnets AND the 4 containers in Step 2 (idempotent)

# Verify both networks exist with exact CIDRs
docker network inspect acme-main acme-ops \
  --format '{{.Name}}: {{range .IPAM.Config}}{{.Subnet}} (gw {{.Gateway}}){{end}}'

Expected Output:

acme-main: 10.10.0.0/24 (gw 10.10.0.1)
acme-ops: 10.10.1.0/28 (gw 10.10.1.1)

Notes:

In any standard IPv4 subnet (like 10.10.0.0/24), two IP addresses are reserved and cannot be assigned to individual devices, but they aren't for the gateway:

  • Network Address: 10.10.0.0 — identifies the network itself.
  • Broadcast Address: 10.10.0.255 — used to send data packets to every device on this subnet simultaneously.

Where does the Gateway go?

The default gateway (your router) is actually a standard host device on the network. It requires an assignable host IP address. By convention/best practice, network administrators typically assign the default gateway to either:

  • The first usable IP: 10.10.0.1
  • The last usable IP: 10.10.0.254

Quick Summary Table for 10.10.0.0/24

Address Type Assignable to a device?
10.10.0.0 Network Identifier ❌ Reserved
10.10.0.1 First Usable IP (often assigned to Gateway) ✅ Yes
10.10.0.2 – 10.10.0.254 Usable Host IPs ✅ Yes
10.10.0.255 Broadcast Address ❌ Reserved
  • Gateway (10.10.0.1): Acts as the default router for any container on the acme-main network. When a container needs to talk to the outside world or another network, it sends traffic here.
  • Usable Container IPs (10.10.0.2 through 10.10.0.254): These are the addresses Docker can dynamically assign or statically bind to your containers.
  • Network (10.10.0.0) & Broadcast (10.10.0.255): Still reserved and blocked from assignment, just as discussed.

Step 2: Launch vpc-router, probe, registry, and echo

task lab01:up already launched all four — what each one twins:

Container Address Role GCP twin
vpc-router 10.10.0.250 (main) + 10.10.1.2 (ops), dual-homed Router between subnets, ip_forward=1, hosts the firewall Cloud Router + VPC firewall
probe 10.10.1.3 Ops bastion — your "corporate HQ" client Bastion host / on-prem client
registry 10.10.0.200 Private image registry Artifact Registry (Door 1)
echo 10.10.0.201 nginx logging client source IPs Test backend behind the firewall

Verify the router has two interfaces (eth0 on 10.10.0.250/24 and eth1 on 10.10.1.2/28):

docker exec vpc-router ip -br addr

Step 3: Observe Asymmetric Routing Failure, Then Wire Next-Hop Routes

Right now, probe (10.10.1.3) only knows its default gateway (10.10.1.1). If it pings echo (10.10.0.201), the packet goes to Docker's bridge gateway instead of vpc-router.

sequenceDiagram
    autonumber
    participant P as probe · 10.10.1.3 (acme-ops)
    participant R as vpc-router
    participant E as echo · 10.10.0.201 (acme-main)
    participant G as docker0 gw · 10.10.0.1

    rect rgb(255, 235, 235)
        Note over P,G: PHASE 1 — only the forward route exists ⇒ asymmetric path
        P->>R: ping request — dst 10.10.0.201 via next-hop 10.10.1.2
        R->>E: forwarded out eth0 (10.10.0.250)
        Note over E,G: echo has no route to 10.10.1.0/28,<br/>so its reply falls back to the default route
        E->>G: reply sent to default gw 10.10.0.1
        G--xP: ✗ lost — docker bridges are isolated
        Note over P: request arrived, reply never returned → ping timeout
    end

    rect rgb(232, 245, 233)
        Note over P,G: PHASE 2 — BOTH directions routed ⇒ symmetric path
        P->>R: ping request — dst 10.10.0.201 via next-hop 10.10.1.2
        R->>E: forwarded out eth0 (10.10.0.250)
        Note over E: return route added: 10.10.1.0/28 via 10.10.0.250
        E->>R: reply sent to next-hop 10.10.0.250
        R->>P: forwarded out eth1 → ping replies ✓
    end

Run this hands-on proof:

# 1. Add ONLY the forward route on probe:
docker exec probe ip route add 10.10.0.0/24 via 10.10.1.2

# 2. Try to ping echo (Fails! Because echo doesn't know how to reach 10.10.1.0/28)
docker exec probe ping -c 1 -W 2 10.10.0.201 || echo "EXPECTED FAIL: Missing return route on echo"

# 3. Now add the return route on echo and test again:
docker exec echo ip route add 10.10.1.0/28 via 10.10.0.250
docker exec probe ping -c 2 10.10.0.201
docker exec probe curl -sI http://10.10.0.201 | head -n 1

That manual dance was the lesson — from now on, task lab01:routes applies the same end state idempotently.


Step 4: Enforce a Stateful Firewall on vpc-router (GCP VPC Firewall Twin)

By default, Linux FORWARD policy is ACCEPT (everything talks to everything). On GCP, VPC ingress is deny-all by default, and you open specific ports via google_compute_firewall. Let's configure vpc-router to match GCP:

task lab01:firewall   # FORWARD policy DROP + conntrack ESTABLISHED,RELATED + ICMP allow
                      # (order matters: routes from Step 3 first, THEN close the door)

# 4. Verify: Ping works, but HTTP (port 80) to echo is BLOCKED!
docker exec probe ping -c 1 10.10.0.201
docker exec probe curl -sS -m 3 -o /dev/null -w 'HTTP Code: %{http_code}\n' http://10.10.0.201 || echo "Blocked by firewall as expected"

# 5. Inspect packet drop counters on vpc-router
docker exec vpc-router iptables -L FORWARD -n -v --line-numbers

Step 5: Your Reusable routes.sh

Because container routing tables live in volatile kernel memory, restarting a container or Minikube node wipes manual routes. The idempotent fix-it script is vendored at ./acme-k8s-lab/routes.sh — open it and read the three route blocks (probe forward-route, helper return-routes, Minikube-node return-routes selected by the name.minikube.sigs.k8s.io=acme label). Run it via task routes after every restart — this is triage signature #03.


4. Deliberate Break & Triage Drills

Run each break, observe the exact symptom, and restore:

Drill 1: The Stateless Firewall Trap (Missing ESTABLISHED,RELATED)

# Temporarily allow port 80 from probe -> echo, but DELETE the conntrack rule (Rule #1)
docker exec vpc-router iptables -A FORWARD -s 10.10.1.3 -d 10.10.0.201 -p tcp --dport 80 -j ACCEPT
docker exec vpc-router iptables -D FORWARD 1

# Test: SYN goes through, echo sends SYN-ACK, router DROPS the SYN-ACK!
docker exec probe curl -sS -m 3 http://10.10.0.201 || echo "Timed out - Return SYN-ACK dropped"

# Restore conntrack at Rule #1 and remove temporary port 80 rule:
docker exec vpc-router iptables -I FORWARD 1 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
docker exec vpc-router iptables -D FORWARD -s 10.10.1.3 -d 10.10.0.201 -p tcp --dport 80 -j ACCEPT

5. Readiness Gate G-01

Before moving to Lab 02, verify you can answer without looking:

  1. Why did ping fail when only probe had a route to 10.10.0.0/24? (Packets must route in both directions; echo sent replies to 10.10.0.1 instead of 10.10.0.250).
  2. What is the difference between Connection refused and Connection timed out (000) across a firewall? (DROP discards the SYN silently → client waits for timeout; REJECT or a closed port with an open firewall sends TCP RST → immediate Connection refused).

Next Step: Lab 02 · Minikube Cluster, IPAM & Cilium CNI Dataplane →