Train Your Architect Brain: In interviews, design reviews, and production incidents, knowing how to run a command is 20% of the job; knowing why an boundary exists, what fails when it's misconfigured, and when to choose an alternative is the other 80%.
Pod and Service CIDRs are baked in at cluster creation and painful to change later. Always allocate four distinct address classes before touching Docker, Minikube, or Terraform.
Node Capacity Is Often Capped by Pod CIDR, Not Node Subnet: A /20 Pod CIDR has sixteen /24 blocks. With default Kubernetes settings (--node-cidr-mask-size=24, up to 110 pods/node), your cluster cannot scale past 16 nodes even if your node subnet has 250 free IPs! On GKE, setting max_pods_per_node = 64 shrinks the slice to /25 (128 IPs), doubling max nodes to 32.
Overlap Is Silent Until You Connect Networks: Two clusters with 10.20.0.0/20 run fine in isolation. The day you peer their VPCs or attach them to Cloud Router/VPN, routing breaks catastrophically.
Never Burn 10.x on Managed Databases: Use RFC 6598 (100.64.0.0/10) for Private Services Access (Cloud SQL) so enterprise 10.x ranges remain clean for VPCs and on-prem spokes.
When a second cluster is justified (see ADR-01), each region gets a self-contained slice of the blueprint — third-octet = region index for nodes, +10 for pods, +20 for services. Allocate all rows before peering.
Region
Profile
Node Subnet
Pod CIDR
Service CIDR
Cilium cluster.id
Domain
us (hub)
acme
10.10.0.0/24 (acme-main)
10.20.0.0/20
10.30.0.0/24
1
orders
eu
acme-eu
10.11.0.0/24 (acme-eu)
10.21.0.0/20
10.31.0.0/24
2
payments
asia
acme-asia
10.12.0.0/24 (acme-asia)
10.22.0.0/20
10.32.0.0/24
3
inventory
All acme-* subnets live inside 10.8.0.0/13 — one route per node sends every region through its local vpc-router leg (10.1x.0.250), the NCC hub-spoke twin.
When regulatory compliance (PCI-DSS CDE vs non-CDE) mandates separate VPC firewall tables or teams require independent GKE upgrade cadences. (Compromise: Per-domain node pools with dedicated pod secondary ranges in 1 cluster).
Works for legacy apps that only log to files; uses Pod's Workload Identity.
Costs ~32Mi RAM/pod; breaks under Default-Deny Egress unless collector/API port + metadata server (169.254.169.252:988) are allowed!
Pattern C: Loopback OTLP + otel-collector Sidecar
App sends traces/logs/metrics to 127.0.0.1:4318; OTel batches, redacts PII, exports to backend.
Vendor-neutral (swap debug exporter for googlecloud with 1 line); offloads batching/retries from app.
Costs ~64Mi RAM/pod; requires egress hole to telemetry backend.
ADR-05: Ownership Boundary — Terraform vs FluxCD Handshake¶
Rule: Terraform never runs kubectl apply or uses the Kubernetes/Helm Terraform providers.
Why?
Security: GitLab CI runners never need network access to the private GKE control plane (10.10.250.0/28). Flux pulls from inside the cluster out to GitLab over Door 3 (Cloud NAT).
State Reliability: Broken Kubernetes CRDs inside Terraform state wedge terraform plan/destroy. Separating infrastructure (Terraform) from workloads (Flux) via deterministic names (orders-app@<project>.iam.gserviceaccount.com) eliminates circular dependencies.
ADR-06: The dom-data Tier — One Shared DB (orders+inventory), One Private DB (payments)¶
Rule: All databases live in a dedicated dom-data namespace, outside every domain's trust boundary and outside the Istio mesh (no sidecar injection). orders and inventory share commerce-db (schemas sales + inventory); payments owns payments-db alone.
Why the shared schema for orders+inventory?
Atomicity: "place order = decrement stock + insert order row" must be one ACID transaction. An HTTP POST /reserve hop between services cannot share a transaction boundary — you'd need two-phase commit for what a BEGIN…COMMIT gives free.
Honest trade-off (shared fate):orders_rw is granted UPDATE on inventory.items but notINSERT/DELETE — the shared-schema tension is intentional and debatable: coupling vs consistency. When commerce-db dies, both domains degrade — that blast radius is the lesson, not an accident.
Why payments does NOT share: payment records are the highest-sensitivity data; a separate instance means a compromised orders_rw role cannot read card flow, and orders → payments-db:5432 is verifiably dropped at L4.
Why no mesh on dom-data: Istio AuthorizationPolicy enforces at the server-side Envoy — DB pods have none, and pgwire has no HTTP semantics anyway. The data tier is protected by NetworkPolicy (L4) alone; on GCP the identity gap is closed by Cloud SQL Auth Proxy + IAM DB auth, not by mesh mTLS.