Learning platform · GCP free tier

Acme Labs
One GKE cluster, three domains, enterprise-grade plumbing

Hand-written Terraform networking · GitLab 3-tier groups · Workload Identity Federation · Flux + Helm · sidecars → central Cloud Logging
Arrow keys / click to navigate · P to print · companion file: acme-labs-plan.md

What we are building, and what we are deliberately not

Minimal footprint, maximum exposure to the hidden networking and identity mechanics

Build

  • 1 private zonal GKE cluster (Dataplane V2)
  • 3 domains: orders (Go), payments (Python), inventory (Node)
  • Per pod: app + Envoy + OTel collector + Fluent Bit
  • 1 central log bucket, one Log Router sink
  • Custom VPC with real IPAM, 4 egress doors
  • GitLab CI → GCP via no-key federation
  • Flux + Helm for everything in-cluster

Design principles

  • Terraform owns GCP; Flux owns Kubernetes
  • Shared code lives in Tier 1 modules (versioned)
  • Secure by default: deny first, then open doors
  • Identities are deterministic names, not lookups
  • Build with training wheels, then remove them (P8)
  • Every phase ends with a predict-then-verify gate

Not now (stretch menu)

  • Cloud SQL via PSA (100.64/20)
  • Hybrid HA VPN + NCC-style site
  • Per-domain node pools / Shared VPC
  • Consul contract bus + self-hosted runner
  • Cloud Storage / Firebase / Pub/Sub via Tier-1 modules
Cost: a few $/day while up, ≈ $0 destroyed. Budget alarm first.

Decision: can ONE cluster serve three isolated domains? Yes.

A cluster lives in one VPC — so isolation is layered, not VPC-per-domain. "3 VPNs" becomes 3 isolation domains.
LayerMechanismStops
Cloud IAM1 GSA per domain (Workload Identity)Borrowing another domain's cloud rights
L3/L4NetworkPolicy default-deny (Dataplane V2)Un-listed pod→pod paths
L7 identityIstio mTLS + AuthorizationPolicy (SPIFFE SA)Right path, wrong caller
CapacityResourceQuota / LimitRangeNoisy neighbours
VPC (stretch)Node pool per domain with its own pod rangeLets VPC firewalls see domains

Why one cluster is right for the lab

  • Free-tier management-fee credit covers one zonal cluster
  • You still touch every network concept (ranges, NAT, routes, DNS, LB)
  • Three clusters would triple cost and teach little extra

When you would need more

  • Hard regulatory network separation → separate clusters / Shared VPC with 3 subnets (S3, S6)
  • Different upgrade cadences or blast radius

Target architecture (np)

Everything inside one VPC acme-main; the control plane is reachable only from your admin /32
GCP project my-devops-journey-502420 (np) VPC acme-main · custom mode · default route deleted at birth Subnet gke-main 10.10.0.0/24 · pods 10.20.0.0/20 · svcs 10.30.0.0/24 → GKE acme-np (private, zonal, Dataplane V2) ns dom-orders · Go ns dom-payments · Python ns dom-inventory · Node Pod app istio-proxy otel-collector fluent-bit GSA <domain>-app (WI)default-deny + allows5th: cloud-sql-proxy → Cloud SQL Platform namespaces (Flux-managed) flux-system · istio-system (istiod) · gateway (gke-l7-rilb → proxy-only 10.10.254.0/23) · kube-system · dom-data (Postgres → Cloud SQL twin) Router + NAT (2 static IPs) · routes · private DNS zones Firewall: IAP ssh · LB health · proxy→nodes · master→webhooks Cloud Logging_Default → sink → central-app-logs Artifact Registryapps + dockerhub-mirror Cloud SQL · PSA 100.64.0.0/20commerce-sql ← orders + inventory (shared)payments-sql ← payments only (private) Trace · Monitoringvia OTel googlecloud exporter GitLab.comFlux source · CI + OIDC (via NAT) GCS state bucketsnp / p tfstate · versionedProd project: state + SAs only

Each domain, up close — same pod shape, different stack and data

One Tier-2 chart, three releases. What differs per domain: language, endpoints, which database, and which calls are allowed out.
ns dom-orders · Go public API · the saga driver Pod (5 containers) app :8080 — POST /orders, GET /orders/:id istio-proxy otel-collector fluent-bit sql-proxy KSA orders → GSA orders-app (WI) envFrom: db-creds → DATABASE_URL default-deny + egress :8080/:5432/:53 commerce-db :5432 (shared) decrement stock + order row — ONE tx → inventory.dom-inventory GET /items/:id (mTLS) → payments.dom-payments POST /charge (mTLS) → declines: compensating tx restores stock ✗ payments-db:5432 — DROPPED by NetPol
ns dom-payments · Python internal · FastAPI, own private DB Pod (5 containers) app :8080 — POST /charge istio-proxy otel-collector fluent-bit sql-proxy KSA payments → GSA payments-app (WI) envFrom: db-creds → DATABASE_URL default-deny + egress :5432/:53 payments-db :5432 (private) payments rows · 402 on simulate=fail/.99 ingress: SPIFFE orders-sa only (mTLS) denied-in: every other principal → 403 RBAC ✗ → inventory — DROPPED (egress+ingress) ✗ → commerce-db:5432 — DROPPED
ns dom-inventory · Node internal · Express catalog on shared DB Pod (5 containers) app :8080 — GET /items, /items/:id istio-proxy otel-collector fluent-bit sql-proxy KSA inventory → GSA inventory-app (WI) envFrom: db-creds → DATABASE_URL default-deny + egress :5432/:53 commerce-db :5432 (shared) owns inventory schema · reads catalog ingress: SPIFFE orders-sa only (mTLS) shares DB w/ orders — same failure domain ✗ → payments — DROPPED (egress+ingress) ✗ → payments-db:5432 — DROPPED
Local lab: the two purple boxes are Postgres StatefulSets in dom-data (no sidecar — L4 NetPol only). GCP: commerce-sql + payments-sql Cloud SQL via PSA, and sql-proxy is the 5th sidecar that puts WI identity inside the DB handshake.

GitLab 3-tier layout — folder path is a permission boundary

Same shape as the corporate model, learning names. Variables and federation trust inherit from the root group.
Tier 1 · Platform

platform/*

  • bootstrap
  • network
  • gke-platform
  • observability
  • terraform-modules ★
  • ci-templates ★
  • contract-bus (S4)
★ shared code lives here — never copied downstream. Analogy: the power grid & roads
Tier 2 · Shared + GitOps

domain-shared/*

  • shared-services (SQL, Pub/Sub later)
  • configurations/helm-charts
  • configurations/flux-releases
Analogy: shared utilities & blueprints
Tier 3 · Applications

applications/<domain>

  • orders/ → orders-svc, orders-infra
  • payments/ → payments-svc, payments-infra
  • inventory/ → inventory-svc, inventory-infra
AR_REPO variables exist only here. Analogy: individual storefronts
Hands-on proof: create a repo outside applications/ → image push fails (variable not inherited). Create a project outside the root group → WIF attribute condition rejects the token.

Step 0 — the starting point: repos & folders

Two homes: the labs' repo + your scratch dir locally, one GitLab project per box on GCP. Names are deterministic — nothing is ever looked up.
GitLab projects (create once)
gitlab.com/acme-labs-<handle>/
├─ platform/            ← Tier 1: infra lives here
│   bootstrap · network · gke-platform
│   observability · terraform-modules ★ · ci-templates ★
├─ domain-shared/       ← Tier 2: shared utilities
│   shared-services        (Cloud SQL: commerce-sql, payments-sql)
│   configurations/helm-charts   (acme-microservice, OCI-published)
│   configurations/flux-releases (GitOps repo Flux watches)
└─ applications/        ← Tier 3: storefronts
    orders/    orders-svc + orders-infra
    payments/  payments-svc + payments-infra
    inventory/ inventory-svc + inventory-infra
★ shared code — consumed by git tag (?ref=v…), never copied downstream
Local folders (labs)
KubernetesConcepts/     ← this repo
├─ apps/orders · payments · inventory
├─ apps/data/      (dom-data: DBs + seed)
├─ charts/acme-microservice/
└─ src/1 … src/12  (concept manifests)

~/acme-k8s-lab/         ← scratch (never committed)
├─ charts/   (the chart you author, Lab 03)
├─ gitops/   clusters/local + infrastructure/
│            {controllers,configs,data} + apps/local
└─ scripts/  cluster-up · down · verify-all
Lab 10 proves it: delete everything under ~/acme-k8s-lab, rebuild green from the repo + scripts
Two environments, one shape
np = my-devops-journey-502420 → cluster acme-np, SA pipeline-np, Flux clusters/np — the labs build this
prod = acme-labs-p-<suffix> → state buckets + SAs only (pipeline-p protected-main / pipeline-p-ro), Flux clusters/p — deploys gated by WIF CEL on the protected branch

Identity: GitLab → GCP with no keys, three trust levels

Trust is encoded in a CEL attribute mapping — the "protected main only" rule lives in Google, not GitLab
1 · GitLab jobid_tokens → signed OIDC JWT (claims: project_path, ref, ref_protected)
→
2 · STSWorkload Identity Pool gitlab-pool + provider checks issuer & attribute condition
→
3 · ImpersonateprincipalSet must hold workloadIdentityUser on the target SA
→
4 · Terraformshort-lived token via ADC; GCS backend + providers
Service accountProjectWho can impersonate (principalSet)Power
pipeline-npnproot_group = acme-labs-…Edit + IAM admin (any branch)
pipeline-p-roprodroot_group = acme-labs-…Viewer — feature branches get a read-only plan
pipeline-pprodroot_group_ref = acme-labs-…:main:trueEdit — protected main only
No Organization? A personal Google account has no folders or org policies, so the "corporate folder" level is modelled by two projects (np, p). With an org, add google_folder and the same SAs move to folder-level IAM.

IPAM: four kinds of address space, four different reasons

Decide the table before writing Terraform — pod/service ranges are hard to change later
1 · Primary / routable

Nodes 10.10.0.0/24 · ops 10.10.1.0/28 · master 10.10.250.0/28

  • The only space you'd ever advertise to on-prem/VPN
  • Must never overlap anything you may later connect
  • Master range: mandatory /28 for the private control plane
2 · Pods & Services

gke-pods 10.20.0.0/20 · gke-svcs 10.30.0.0/24

  • Secondary ranges on gke-main (VPC-native alias IPs)
  • max_pods_per_node=64 → /25 per node → up to 32 nodes
  • Hybrid: pod IPs unknown to on-prem → masquerade (S2)
3 · Private Services Access

100.64.0.0/20 (stretch S1)

  • Peered range for Cloud SQL via servicenetworking
  • RFC 6598 space: doesn't burn scarce 10.x
  • Peering is non-transitive
4 · Proxy-only

proxy-only-subnet 10.10.254.0/23

  • purpose = REGIONAL_MANAGED_PROXY
  • Google's Envoy fleet for internal L7 LBs / gke-l7-rilb
  • Your VMs/pods never get an IP from it — but firewalls must allow it to your nodes

Why the default internet route is deleted — and the four outbound doors

Lab: start with a "training wheels" default route, then remove it (P8) and repair every break

VPC acme-main

  • GKE nodes & pods
  • probe VM (ops)
DELETED ON CREATE
0.0.0.0/0 → default-internet-gateway
Unknown IP ⇒ packet dropped (no route)
Most-specific prefix wins; tags scope which instances a route applies to.
Door 1 · Google APIs PGA + private VIP
Subnet private_ip_google_access=true + route 199.36.153.8/30 + private DNS (*.googleapis.com / gcr.io / pkg.dev → private.googleapis.com). Logging, GCS, Artifact Registry.
Door 2 · /32 allow-list Explicit routes, priority 900
One inet-<name> route per approved external IP (map variable vip_allowlist).
Door 3 · Cloud NAT + tag Route prio 999 for tag allowed-open-nat
Two static NAT IPs so partners can allow-list you. NAT only translates traffic whose next hop is the internet gateway. Tags are node-level — pods inherit.
Door 4 · Hybrid (stretch S2) HA VPN + Cloud Router → onprem-sim VPC
BGP routes, DNS peering, ip-masq-agent for pod traffic. Stands in for NCC/on-prem.

Sidecars → one central Cloud Logging

Three delivery patterns for the same goal; one Log Router sink collects the three domains
Pod (one per domain) appJSON stdout + /var/log/app istio-proxymTLS · JSON access log otel-collectorOTLP → googlecloud fluent-bittail file → stackdriver emptyDir shared volume (app ⇄ fluent-bit) Auth to Cloud Logging API: Workload Identity KSA → GSA (logging.logWriter, trace.agent) Needs egress to metadata server 169.254.169.252:988 + Google APIs (Door 1) when default-deny is on Pattern Astdout → GKE node agent Pattern Bfile-tail sidecar → API Pattern COTLP sidecar → logs/traces/metrics Cloud Logging · _Defaultexclusion for ^dom- (avoid double storage) Log Router sink → central-app-logsfilter: k8s_container ∧ namespace =~ ^dom-30d retention · Log Analytics (SQL) onlog-based metric: domain_errors{domain}

Domains talk to each other — only the way you allow

Two independent guards; each fails with a different signature (timeout vs 403)
InternalGateway orders (Go)dom-orders payments (Py)dom-payments inventory (Node)dom-inventory × payments ↔ inventory: denied by design

Guard 1 · NetworkPolicy (L3/4)

default-deny both directions + explicit allows. Egress must also allow DNS, istiod (15012), GKE metadata server (169.254.169.252:988) — or Workload Identity silently breaks.

Guard 2 · Istio AuthorizationPolicy (L7)

payments accepts only cluster.local/ns/dom-orders/sa/orders over STRICT mTLS. Failure signature: HTTP 403 (RBAC).

Why both

Network path ≠ caller identity. Remove guard 1 and test: you now see guard 2 alone — and learn to read each layer's error.

Flux + Helm: implementation order and why

Kustomization dependsOn + wait: true build the layers; nothing is applied by hand after bootstrap
0 · Cluster (Terraform)nodes Ready, kubectl works from admin /32
→
1 · flux bootstrap→ flux-releases repo, path clusters/np
→
2 · infra-controllersSources → istio-base (CRDs) → istiod (webhook healthy)
→
3 · infra-configsNamespaces (injection label), mTLS STRICT, default-deny, quotas, gateway
→
4 · apps3 HelmReleases from OCI chart acme-microservice

Why this order

  • CRDs before CRs; webhook before pods — or sidecars silently don't inject
  • Namespace label must pre-exist the first pod
  • Security baseline lands before any workload

Terraform ⇄ Flux handshake

  • TF creates GSA orders-app + WI binding for dom-orders/orders
  • Chart annotates the KSA with that same predictable email
  • CI never touches the Kubernetes API → control plane stays locked down
  • Images: CI → Artifact Registry → tag bump commit → Flux

Roadmap with readiness gates

Each gate = predict the outcome first, then verify. Roughly 5–7 weeks at several hours/week.
PhaseWhatHidden nuance revealedGate
P0–P1Groups, bootstrap (projects, state, WIF, SAs)Attribute mapping = trust boundaryG1
P2CI templates; prod blocked from feature branchesProtected-ref federationG2
P3Network v1: IPAM, routes, NAT, DNS, firewallNAT needs IGW next hop; tags are node-levelG3
P4Private GKE, node SA, Artifact Registry + mirrorPrivate endpoint vs authorized networksG4
P5Flux bootstrap, Istio via dependsOnWebhook ordering, master→node portsG5
P6Central log bucket, sink, sidecar configsThree log delivery patternsG6
P73 domains: chart, infra modules, policiesDefault-deny breaks WI/DNS/xDSG7
P8Delete default route; repair each flowWhich door each packet really usesG8
P9Internal Gateway (proxy-only subnet)STRICT mTLS vs LB plaintext, health-check rangesG9
P10Stretch: PSA/Cloud SQL, hybrid VPN, pool-per-domain, Consul …PSA non-transitive, masquerade—

Growing the platform later without copy-paste

Add Cloud Storage, Firebase, Pub/Sub, Cloud SQL as Tier-1 modules consumed at Tier 3

1 · Write once (Tier 1)

  • terraform-modules/modules/gcs-bucket
  • Secure defaults baked in (uniform access, public-access prevention)
  • Release a tag: v0.2.0

2 · Consume (Tier 3)

  • module "uploads" { source = "...//modules/gcs-bucket?ref=v0.2.0" }
  • Grant the domain GSA writer access
  • State is per project — blast radius stays small

3 · Wire to the app

  • Deterministic bucket name into HelmRelease values
  • Workload Identity already gives the pod access
  • Update NetworkPolicy/doors only if a new destination appears
Hidden-nuance checklist (tick only what you've observed): NAT/IGW next hop · PGA vs private VIP · node-level tags · master→webhook ports · secondary-range sizing · proxy-only subnet firewalls · health-check ranges · default-deny breaks WI · STRICT mTLS vs plaintext · NAT port exhaustion · PSA non-transitive · create-time route deletion · WIF attribute conditions