Per pod: app + Envoy + OTel collector + Fluent Bit
1 central log bucket, one Log Router sink
Custom VPC with real IPAM, 4 egress doors
GitLab CI → GCP via no-key federation
Flux + Helm for everything in-cluster
Design principles
Terraform owns GCP; Flux owns Kubernetes
Shared code lives in Tier 1 modules (versioned)
Secure by default: deny first, then open doors
Identities are deterministic names, not lookups
Build with training wheels, then remove them (P8)
Every phase ends with a predict-then-verify gate
Not now (stretch menu)
Cloud SQL via PSA (100.64/20)
Hybrid HA VPN + NCC-style site
Per-domain node pools / Shared VPC
Consul contract bus + self-hosted runner
Cloud Storage / Firebase / Pub/Sub via Tier-1 modules
Cost: a few $/day while up, ≈ $0 destroyed. Budget alarm first.
Decision: can ONE cluster serve three isolated domains? Yes.
A cluster lives in one VPC — so isolation is layered, not VPC-per-domain. "3 VPNs" becomes 3 isolation domains.
Layer
Mechanism
Stops
Cloud IAM
1 GSA per domain (Workload Identity)
Borrowing another domain's cloud rights
L3/L4
NetworkPolicy default-deny (Dataplane V2)
Un-listed pod→pod paths
L7 identity
Istio mTLS + AuthorizationPolicy (SPIFFE SA)
Right path, wrong caller
Capacity
ResourceQuota / LimitRange
Noisy neighbours
VPC (stretch)
Node pool per domain with its own pod range
Lets VPC firewalls see domains
Why one cluster is right for the lab
Free-tier management-fee credit covers one zonal cluster
You still touch every network concept (ranges, NAT, routes, DNS, LB)
Three clusters would triple cost and teach little extra
When you would need more
Hard regulatory network separation → separate clusters / Shared VPC with 3 subnets (S3, S6)
Different upgrade cadences or blast radius
Target architecture (np)
Everything inside one VPC acme-main; the control plane is reachable only from your admin /32
Each domain, up close — same pod shape, different stack and data
One Tier-2 chart, three releases. What differs per domain: language, endpoints, which database, and which calls are allowed out.
Local lab: the two purple boxes are Postgres StatefulSets in dom-data (no sidecar — L4 NetPol only). GCP: commerce-sql + payments-sql Cloud SQL via PSA, and sql-proxy is the 5th sidecar that puts WI identity inside the DB handshake.
GitLab 3-tier layout — folder path is a permission boundary
Same shape as the corporate model, learning names. Variables and federation trust inherit from the root group.
Root group: gitlab.com/acme-labs-<handle>
Automatic inheritance bus: WIF identity (attribute.root_group) · group CI variables (WIF_PROVIDER, SA_NP, state buckets) · protected-only prod variables
Tier 1 · Platform
platform/*
bootstrap
network
gke-platform
observability
terraform-modules ★
ci-templates ★
contract-bus (S4)
★ shared code lives here — never copied downstream. Analogy: the power grid & roads
Tier 2 · Shared + GitOps
domain-shared/*
shared-services (SQL, Pub/Sub later)
configurations/helm-charts
configurations/flux-releases
Analogy: shared utilities & blueprints
Tier 3 · Applications
applications/<domain>
orders/ → orders-svc, orders-infra
payments/ → payments-svc, payments-infra
inventory/ → inventory-svc, inventory-infra
AR_REPO variables exist only here. Analogy: individual storefronts
Hands-on proof: create a repo outside applications/ → image push fails (variable not inherited). Create a project outside the root group → WIF attribute condition rejects the token.
Step 0 — the starting point: repos & folders
Two homes: the labs' repo + your scratch dir locally, one GitLab project per box on GCP. Names are deterministic — nothing is ever looked up.
Lab 10 proves it: delete everything under ~/acme-k8s-lab, rebuild green from the repo + scripts
Two environments, one shape
np = my-devops-journey-502420 → cluster acme-np, SA pipeline-np, Flux clusters/np — the labs build this prod = acme-labs-p-<suffix> → state buckets + SAs only (pipeline-p protected-main / pipeline-p-ro), Flux clusters/p — deploys gated by WIF CEL on the protected branch
Identity: GitLab → GCP with no keys, three trust levels
Trust is encoded in a CEL attribute mapping — the "protected main only" rule lives in Google, not GitLab
3 · ImpersonateprincipalSet must hold workloadIdentityUser on the target SA
→
4 · Terraformshort-lived token via ADC; GCS backend + providers
Service account
Project
Who can impersonate (principalSet)
Power
pipeline-np
np
root_group = acme-labs-…
Edit + IAM admin (any branch)
pipeline-p-ro
prod
root_group = acme-labs-…
Viewer — feature branches get a read-only plan
pipeline-p
prod
root_group_ref = acme-labs-…:main:true
Edit — protected main only
No Organization? A personal Google account has no folders or org policies, so the "corporate folder" level is modelled by two projects (np, p). With an org, add google_folder and the same SAs move to folder-level IAM.
IPAM: four kinds of address space, four different reasons
Decide the table before writing Terraform — pod/service ranges are hard to change later
The only space you'd ever advertise to on-prem/VPN
Must never overlap anything you may later connect
Master range: mandatory /28 for the private control plane
2 · Pods & Services
gke-pods 10.20.0.0/20 · gke-svcs 10.30.0.0/24
Secondary ranges on gke-main (VPC-native alias IPs)
max_pods_per_node=64 → /25 per node → up to 32 nodes
Hybrid: pod IPs unknown to on-prem → masquerade (S2)
3 · Private Services Access
100.64.0.0/20 (stretch S1)
Peered range for Cloud SQL via servicenetworking
RFC 6598 space: doesn't burn scarce 10.x
Peering is non-transitive
4 · Proxy-only
proxy-only-subnet 10.10.254.0/23
purpose = REGIONAL_MANAGED_PROXY
Google's Envoy fleet for internal L7 LBs / gke-l7-rilb
Your VMs/pods never get an IP from it — but firewalls must allow it to your nodes
Why the default internet route is deleted — and the four outbound doors
Lab: start with a "training wheels" default route, then remove it (P8) and repair every break
VPC acme-main
GKE nodes & pods
probe VM (ops)
DELETED ON CREATE 0.0.0.0/0 → default-internet-gateway Unknown IP ⇒ packet dropped (no route)
Most-specific prefix wins; tags scope which instances a route applies to.
Door 1 · Google APIsPGA + private VIP Subnet private_ip_google_access=true + route 199.36.153.8/30 + private DNS (*.googleapis.com / gcr.io / pkg.dev → private.googleapis.com). Logging, GCS, Artifact Registry.
Door 2 · /32 allow-listExplicit routes, priority 900 One inet-<name> route per approved external IP (map variable vip_allowlist).
Door 3 · Cloud NAT + tagRoute prio 999 for tag allowed-open-nat Two static NAT IPs so partners can allow-list you. NAT only translates traffic whose next hop is the internet gateway. Tags are node-level — pods inherit.
Door 4 · Hybrid (stretch S2)HA VPN + Cloud Router → onprem-sim VPC BGP routes, DNS peering, ip-masq-agent for pod traffic. Stands in for NCC/on-prem.
Sidecars → one central Cloud Logging
Three delivery patterns for the same goal; one Log Router sink collects the three domains
Domains talk to each other — only the way you allow
Two independent guards; each fails with a different signature (timeout vs 403)
Guard 1 · NetworkPolicy (L3/4)
default-deny both directions + explicit allows. Egress must also allow DNS, istiod (15012), GKE metadata server (169.254.169.252:988) — or Workload Identity silently breaks.
Guard 2 · Istio AuthorizationPolicy (L7)
payments accepts only cluster.local/ns/dom-orders/sa/orders over STRICT mTLS. Failure signature: HTTP 403 (RBAC).
Why both
Network path ≠ caller identity. Remove guard 1 and test: you now see guard 2 alone — and learn to read each layer's error.
Flux + Helm: implementation order and why
Kustomization dependsOn + wait: true build the layers; nothing is applied by hand after bootstrap
0 · Cluster (Terraform)nodes Ready, kubectl works from admin /32