Lab 19 · Kustomize & Multi-Environment Promotion¶
Core Lesson: You've met the Flux
KustomizationCRD (lab 09). Now meet kustomize the tool — template-free YAML customization: a sharedbase/+ per-envoverlays/that patch replicas, resources, and config without copy-pasting YAML. Then wire it to promotion: the same base flows dev→stage→prod through Flux, each overlay being one auditable diff. GCP Twin: this is exactly the release-repo shape Flux estates use —base/+overlays/<env>reconciled by per-cluster Kustomizations.
1. Architecture Decision: Kustomize vs Helm¶
| Helm | Kustomize | |
|---|---|---|
| Model | templates + values | base YAML + patches |
| Best at | distributing apps (charts, versions, registries) | env promotion of your own manifests — overlays are diffs, not templates |
| Danger | template sprawl | patch drift (patch references a field that moved) |
They compose: Flux renders helmChart then kustomizes the output —
valuesFrom in a HelmRelease is already this idea.
flowchart LR
B[base/<br/>deployment + service<br/>env-agnostic] --> D[overlays/dev<br/>1 replica, cheap]
B --> S[overlays/stage<br/>2 replicas]
B --> P[overlays/prod<br/>4 replicas, PDB, hpa]
D & S & P --> F[Flux Kustomization<br/>per env]
F --> K[(cluster per env)]
2. Step-by-Step Hands-On Build¶
Step 1: The base — deliberately incomplete¶
mkdir -p ~/acme-k8s-lab/promo/{base,overlays/{dev,stage,prod}}
cat << 'EOF' > ~/acme-k8s-lab/promo/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: [deployment.yaml, service.yaml]
commonLabels: {app: orders}
EOF
cat << 'EOF' > ~/acme-k8s-lab/promo/base/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata: {name: orders}
spec:
selector: {matchLabels: {app: orders}}
template:
metadata: {labels: {app: orders}}
spec:
containers:
- name: app
image: orders-api:1.4.0
ports: [{containerPort: 8080}]
EOF
Design rule: the base carries only what's TRUE for every env — never a replica count, never a resource limit, never a hostname.
Step 2: Overlays — each env is a diff, nothing more¶
cat << 'EOF' > ~/acme-k8s-lab/promo/overlays/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: [../../base]
namespace: dev
patches:
- patch: |-
- op: replace
path: /spec/replicas
value: 1
target: {kind: Deployment, name: orders}
images:
- name: orders-api
newTag: dev-latest
EOF
cat << 'EOF' > ~/acme-k8s-lab/promo/overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: [../../base]
namespace: prod
patches:
- patch: |-
- op: replace
path: /spec/replicas
value: 4
target: {kind: Deployment, name: orders}
- patch: |-
- op: add
path: /spec/template/spec/containers/0/resources
value: {requests: {cpu: 200m, memory: 256Mi}, limits: {cpu: "1", memory: 512Mi}}
target: {kind: Deployment, name: orders}
images:
- name: orders-api
newTag: "1.4.0"
EOF
Step 3: kustomize build — promotion previewed before any cluster¶
kustomize build overlays/dev | grep -E "replicas|namespace|image:"
kustomize build overlays/prod | grep -E "replicas|namespace|image:"
diff <(kustomize build overlays/dev) <(kustomize build overlays/prod)
# ← THE answer to "what's different between dev and prod": one diff, auditable in git
Step 4: Wire to Flux — one Kustomization per env¶
# clusters/prod/30-apps.yaml — the prod cluster's Flux Kustomization:
spec:
path: ./promo/overlays/prod # THIS cluster reconciles the prod overlay
# dev cluster's Kustomization points at ./promo/overlays/dev — same repo
Promotion = merge the same base change; each cluster's kustomization picks
it up through its own overlay — the image pin in images: is what makes
prod move only when you choose.
3. Deliberate Break & Triage¶
| Break | Symptom | Lesson |
|---|---|---|
Patch targets name: orders after base renames it |
no resource matches strategic merge patch |
patches are name/namespace/type-coupled — renames cascade |
Put replicas: 1 in the base |
prod overlay inherits it; prod silently runs 1 replica | base stays env-agnostic — the rule that prevents real incidents |
resources: patch missing a required field |
kustomize build fails fast |
build locally in CI — promotion diffs are CI-checkable artifacts |
| Direct-edit a prod namespace in an overlay | kubectl applies to wrong ns | namespace: in overlay pins everything; mixing namespaces in one overlay is a smell |
4. Verify¶
-
diffof dev vs prod builds understood as "the env contract" - You can state kustomize vs helm in the one-table answer
- A promotion traced: base merge → overlay → Flux kustomization → cluster
- You know why
newTagper overlay = the promotion gate
5. Continue — the capstone¶
You now own every piece of the loop. gitlab-spring-app-flux
docs/07-capstone.md strings them together: commit → test → build → scan →
sign → package → gated deploy → canary → rollback — one pipeline, every
phase of this curriculum visible in one file.