Lab 09 · GitOps Ordering with FluxCD: Controllers → Configs → Apps¶
Core Lesson: Replace manual
kubectl applyandhelm upgradecommands with a single FluxCD GitOps repository structured into four strict dependency tiers (10-infra-controllers→20-infra-configs→25-data→30-apps). Prove why removingdependsOnorwait: truecauses the classic GitOps race conditions: CRD-before-CR failure, missing Istio sidecars, and apps starting before their databases exist.
GCP Twin: Phase P5 (flux-releasesrepository,clusters/np).
1. Architecture Decision: Why Order Matters in GitOps¶
If you dump all your Kubernetes YAMLs into a single folder and point Flux or ArgoCD at it, a cold cluster rebuild will fail every time:
| Object | Which Tier Owns It? | What Breaks If It Runs Out of Order? |
|---|---|---|
| Cilium CNI | 0 · Cluster Bootstrap (Outside Flux; GKE Dataplane V2 setting) | Flux controller pods themselves cannot schedule or pull Git repos without a working CNI! |
istio-base, istiod, metallb |
1 · 10-infra-controllers (wait: true) |
If Tier 2 runs before Tier 1 finishes, PeerAuthentication and IPAddressPool fail with no matches for kind ... (CRD not installed). |
Labeled Namespaces (istio-injection: enabled), NetworkPolicy, PeerAuthentication, AuthorizationPolicy |
2 · 20-infra-configs (dependsOn: infra-controllers, wait: true) |
If Tier 3 runs before istiod webhook is ready OR before the Namespace has istio-injection: enabled, application pods start with 1/1 containers (no Envoy sidecar, zero mTLS)! |
dom-data Postgres StatefulSets + db-creds Secrets + db-seed Job |
3 · 25-data (dependsOn: infra-configs, wait: true) |
If Tier 4 runs before this tier, app pods CrashLoopBackOff on cannot reach commerce-db — or worse, pass /readyz against an unseeded database and serve empty catalogs. The seed Job inside the tier makes "ready" mean "schema exists", not just "postgres accepts TCP". |
Application HelmReleases (orders, payments, inventory) |
4 · 30-apps (dependsOn: data) |
Lands last into a fully wired, default-deny, mTLS-enforced namespace whose databases are already seeded. |
2. Step-by-Step Hands-On Build¶
Step 1: Create the Local GitOps Repository Structure (~/acme-k8s-lab/gitops)¶
You can run this either against GitLab.com (flux bootstrap gitlab ...) or 100% locally offline using a local Git repository served inside the cluster (or flux create source git). Below we build the exact directory layout used in GCP Phase P5:
mkdir -p ~/acme-k8s-lab/gitops/{clusters/local,infrastructure/controllers,infrastructure/configs,infrastructure/data,apps/local}
cd ~/acme-k8s-lab/gitops
Step 2: Author the 4 Flux Kustomization Dependency Tiers (clusters/local/)¶
cat << 'EOF' > ~/acme-k8s-lab/gitops/clusters/local/10-infra-controllers.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: infra-controllers
namespace: flux-system
spec:
interval: 5m
path: ./infrastructure/controllers
prune: true
wait: true
timeout: 5m
sourceRef:
kind: GitRepository
name: flux-system
EOF
cat << 'EOF' > ~/acme-k8s-lab/gitops/clusters/local/20-infra-configs.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: infra-configs
namespace: flux-system
spec:
interval: 5m
dependsOn:
- name: infra-controllers
path: ./infrastructure/configs
prune: true
wait: true
timeout: 3m
sourceRef:
kind: GitRepository
name: flux-system
EOF
cat << 'EOF' > ~/acme-k8s-lab/gitops/clusters/local/25-data.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: data
namespace: flux-system
spec:
interval: 5m
dependsOn:
- name: infra-configs
path: ./infrastructure/data
prune: true
wait: true # waits for StatefulSets Ready AND the db-seed Job to Complete
timeout: 5m
sourceRef:
kind: GitRepository
name: flux-system
EOF
cat << 'EOF' > ~/acme-k8s-lab/gitops/clusters/local/30-apps.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps
namespace: flux-system
spec:
interval: 5m
dependsOn:
- name: data # apps land only after databases exist AND are seeded
path: ./apps/local
prune: true
wait: true
timeout: 5m
sourceRef:
kind: GitRepository
name: flux-system
EOF
Step 3: Populate infrastructure/controllers/, infrastructure/configs/, and apps/local/¶
1. infrastructure/controllers/istio.yaml¶
cat << 'EOF' > ~/acme-k8s-lab/gitops/infrastructure/controllers/istio.yaml
apiVersion: v1
kind: Namespace
metadata:
name: istio-system
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
name: istio
namespace: flux-system
spec:
interval: 1h
url: https://istio-release.storage.googleapis.com/charts
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: istio-base
namespace: flux-system
spec:
interval: 10m
targetNamespace: istio-system
chart:
spec:
chart: base
sourceRef:
kind: HelmRepository
name: istio
namespace: flux-system
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: istiod
namespace: flux-system
spec:
interval: 10m
targetNamespace: istio-system
dependsOn:
- name: istio-base
chart:
spec:
chart: istiod
sourceRef:
kind: HelmRepository
name: istio
namespace: flux-system
values:
meshConfig:
accessLogFile: /dev/stdout
accessLogEncoding: JSON
EOF
2. infrastructure/configs/namespaces-and-policies.yaml¶
cat << 'EOF' > ~/acme-k8s-lab/gitops/infrastructure/configs/namespaces-and-policies.yaml
apiVersion: v1
kind: Namespace
metadata:
name: observability
---
apiVersion: v1
kind: Namespace
metadata:
name: dom-orders
labels:
istio-injection: enabled
---
apiVersion: v1
kind: Namespace
metadata:
name: dom-payments
labels:
istio-injection: enabled
---
apiVersion: v1
kind: Namespace
metadata:
name: dom-inventory
labels:
istio-injection: enabled
---
apiVersion: v1
kind: Namespace
metadata:
name: dom-data # deliberately NO istio-injection — the data tier stays outside the mesh (Lab 06, Step 5)
labels:
tier: data
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: acme-internal-pool
namespace: metallb-system
spec:
addresses:
- 10.10.0.240-10.10.0.249
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: acme-l2
namespace: metallb-system
spec:
ipAddressPools:
- acme-internal-pool
---
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
name: default
namespace: dom-payments
spec:
mtls:
mode: STRICT
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: only-orders-sa
namespace: dom-payments
spec:
action: ALLOW
rules:
- from:
- source:
principals: ["cluster.local/ns/dom-orders/sa/orders"]
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: orders-entry # the PUBLIC door (Lab 08): north-south IPs, not SPIFFE
namespace: dom-orders
spec:
action: ALLOW
rules:
- from:
- source:
ipBlocks: ["10.10.1.0/28"]
EOF
3. infrastructure/data/ — copy the repo's data-tier manifests in¶
The Postgres StatefulSets, Secrets, and seed Job already live in this repo (apps/data/). Flux manages them by copying them into the GitOps tree — this is exactly how infrastructure/ in the real flux-releases repo works:
# (dom-data namespace.yml is NOT copied — the infra-configs tier already owns it;
# two Kustomizations must never manage the same object)
cp apps/data/secrets.yml ~/acme-k8s-lab/gitops/infrastructure/data/
cp apps/data/commerce-db/statefulset.yml ~/acme-k8s-lab/gitops/infrastructure/data/commerce-db.yml
cp apps/data/payments-db/statefulset.yml ~/acme-k8s-lab/gitops/infrastructure/data/payments-db.yml
cp apps/data/seed-job.yml ~/acme-k8s-lab/gitops/infrastructure/data/
Ordering nuance Flux gives you for free: the
db-credsSecrets referencedom-orders/dom-payments/dom-inventory, which theinfra-configstier created — one more reasondatamust run afterinfra-configs, not beside it.
Step 4: Bootstrap FluxCD & Watch Ordered Reconciliation¶
Ensure your cluster can reach GitLab (remember Lab 04: acme-m03 has the allowed-open-nat=true default route, or restore default route on nodes for bootstrap):
task lab09:flux # flux install + flux check
# (or `flux bootstrap gitlab ...` with your GITLAB_TOKEN for GitLab-backed parity)
If bootstrapping with GitLab (recommended for GCP P5 parity):
export GITLAB_TOKEN="<your-gitlab-pat>"
flux bootstrap gitlab \
--hostname=gitlab.com \
--owner="acme-labs-<handle>/domain-shared/configurations" \
--repository=flux-releases-local \
--branch=main \
--path=clusters/local \
--token-auth
# Watch the dependency chain converge in order:
flux get kustomizations -w
3. Deliberate Break & Triage Drills (The Point of This Lab!)¶
Drill 1: Remove dependsOn → Trigger the "CRD Not Found" Race Condition¶
If you remove dependsOn: [{name: infra-controllers}] from 20-infra-configs.yaml on a fresh cluster before istio-base or metallb finishes installing CRDs, what happens?
- Exact Error in
flux get kustomizations:
dry-run failed: no matches for kind "IPAddressPool" in version "metallb.io/v1beta1"orno matches for kind "PeerAuthentication" in version "security.istio.io/v1". - Why Flux Self-Heals vs Why
dependsOnIs Still Better: Flux will retry on itsinterval, but withoutdependsOn, workloads may deploy beforePeerAuthenticationorNetworkPolicylands—leaving a security window open during cluster creation!
Drill 2: Remove data from apps.dependsOn → Apps Crash Before Their Database Exists¶
Remove dependsOn: [{name: data}] from 30-apps.yaml and reconcile on a cold cluster:
- Exact symptom:
orderspodsCrashLoopBackOffwithcannot reach commerce-dbinkubectl logs -c app, whileflux get kustomizationsshowsappsracingdata. - The deeper trap: even if postgres pod is
Runningbutdb-seedhasn't finished, apps pass/readyz(TCP connects!) and then serve empty catalogs —wait: trueon the data tier is what makes "ready" mean "seeded", because Flux waits for thedb-seedJob'sCompletecondition, not just StatefulSet readiness.
Drill 3: Test Live Drift Correction (prune: true & Reconciliation)¶
# Manually tamper with a Flux-managed object in the cluster:
kubectl label ns dom-orders tampered=true
# Force Flux reconciliation and verify state management:
flux reconcile kustomization infra-configs --with-source
flux tree kustomization infra-configs
4. Readiness Gate G-09¶
- Why must Cilium CNI be installed before
flux bootstrap, rather than inside10-infra-controllers? (Flux's own pods (source-controller,kustomize-controller,helm-controller) are normal Kubernetes Pods that require Pod IPs and DNS/egress networking to clone the Git repository). - What happens if
wait: trueis omitted on10-infra-controllerseven when20-infra-configshasdependsOn: [{name: infra-controllers}]? (Withoutwait: true, Flux marksinfra-controllersReady the instant theHelmReleaseYAML is accepted by the API server—beforeistiod's webhook pod actually starts running!) - Why does the
datatier sit between configs and apps rather than insideinfra-configsitself? (Configs hold namespaces/policies — instant, idempotent objects. Data holds StatefulSets + a seed Job that takes minutes to converge; keeping it a separateKustomizationwithwait: truegivesappsa "schema exists" gate instead of an "objects applied" gate.)
Next Step: Lab 10 · Disposable Cluster Cold-Rebuild & GKE Production Handoff →
