Skip to content

Stage 4 — The acme-sampleapp sandbox

This is where the isolated exercises stop and the real thing starts. Everything in Stage 2 and Stage 3 teaches one mechanic at a time in a directory you can delete without consequence. The sandbox is five independent Terraform roots that depend on each other through a Consul contract, deploy Helm charts onto a real GKE cluster, and cost real money while they're up.

The single most important difference: no root here can be understood in isolation. backend/infra reads values that cloudsql/infra, infrastructure/infra, and sample-program/infra published. Break one and you learn what a dependency graph actually feels like.

Before you touch it
Source of truth for layout acme-sampleapp-multirepo-sandbox-elaborate/README.md
Source of truth for running it RUNBOOK.md — apply order, Consul dev agent, teardown
Session-by-session history Session log
Concept → file lookup Concept index

This one bills

Stages 1–3 are free or nearly so. The sandbox stands up a GKE cluster, Secret Manager secrets, and optionally a Cloud SQL instance. Read the cost section at the end of the RUNBOOK before enabling anything, and run ./destroy-all.sh when you pause.

The one pattern to read first

sample-program/infra/main.tf is the highest-value file in the sandbox, because it is the for_each-over-modules pattern from the labs applied for real:

module "vpc" {
  source     = "./modules/vpc"
  project_id = var.project_id
  region     = var.region
  subnets    = local.subnets
}

# for_each over a module block — same mechanic from the labs, applied for
# real here: this creates zero, one, or two GKE clusters depending on
# which enable_gke_* flags are true, without duplicating the module call.
module "gke" {
  for_each = local.gke_clusters_enabled
  source   = "./modules/gke"

  project_id   = var.project_id
  zone         = var.zone
  name         = "sample-program-${each.key}"
  network      = module.vpc.network_self_link
  subnetwork   = module.vpc.subnets[each.key].self_link
  machine_type = each.value.machine_type
  node_count   = each.value.node_count
  spot         = each.value.spot
}

module "dns" {
  source     = "./modules/dns"
  project_id = var.project_id
  domain     = var.dns_domain
}

Two things worth sitting with before moving on:

  1. Zero is a legal outcome. If both enable flags are false, local.gke_clusters_enabled is an empty map and module.gke produces nothing at all — no error, no partial cluster. Everything downstream then has to cope with an empty map, which is exactly the guard-propagation bug that cost a real session's debugging time. See Session 2.
  2. The filtering happens in locals, not in the module. The module doesn't know it's conditional. That separation is why the same module is reusable, and why the guard has to be re-applied by every consumer of the filtered map rather than once at the source.

Reference — sandbox README

Sanitized personal practice copy of a real multi-repo GCP/GKE deployment, restructured to mirror the actual project layout: each application owns its own infra/ (and charts/ where applicable), and repos coordinate through Consul-published outputs rather than terraform_remote_state. All org-specific naming has been genericized (davita-transitcare → acme-sampleapp, davita.com → acme.com); GCP is left as-is since that's the actual cloud in use.

The original project's Consul cluster and GCS backends don't exist here, so none of the inherited paths resolve as-written. RUNBOOK.md stands up local substitutes for both — a Consul dev agent and backend "local" — which is what makes this copy runnable against a GCP project of your own.

frontend's app source (Angular/FastAPI) and cloudsql's SQL migrations are intentionally not reconstructed — out of scope for a Terraform-focused sandbox.

Repo layout

sample-program/
  infra/          Terraform: the shared GKE clusters (p/np), VPC, and public
                  DNS zone every other repo depends on. Built as 3 reusable
                  modules (vpc/gke/dns) rather than inline resources.

backend/
  infra/          Terraform: namespace, GSA, Cloud SQL IAM + the shell_script grant
                  bridge, dynamic RBAC, the helm_release that installs the chart below
  charts/acme-sampleapp-backend/   FastAPI + Cloud SQL Auth Proxy sidecar

frontend/
  infra/          Terraform: namespace, GSA, static IP + DNS (vip.tf), the
                  helm_release that installs the chart below
  charts/acme-sampleapp-frontend/  Angular build served by nginx, GCE ingress

cloudsql/
  infra/          Terraform: the actual Cloud SQL instance, IAM-auth enabled,
                  plus the grant/revoke Cloud Functions backend/infra calls into

infrastructure/
  infra/          Terraform: shared SSO secret (Secret Manager), published to
                  every other repo via Consul

This sandbox is runnable — see RUNBOOK.md for the actual apply sequence, a local Consul dev agent, and where to put real GCP values via terraform.tfvars.

How to use this repository for learning

The repository is intentionally split into two layers:

  • 01-basics/ and 02-gcp-terraform/ are small, isolated demonstrations. Use them to learn one Terraform concept at a time without the Acme dependency graph.
  • acme-sampleapp-multirepo-sandbox-elaborate/ is the main project. Its five independent Terraform roots model the organization-style repository boundaries, state ownership, Consul contracts, GKE, Cloud SQL, Helm, and Workload Identity.

Each Acme */infra/ directory owns its own local state file. Do not copy or delete state files between roots. Start a new learning session with ./restore-session.sh: it runs a plan first, reuses existing state, and refuses to create infrastructure if a state file is missing. Use --bootstrap only for the intentional first setup, and --refresh-consul only when the local Consul dev agent was restarted and its in-memory outputs need to be republished. The restore script also verifies that the GKE cluster has a Ready node before waiting on Helm, and supports --repair-failed-helm for a failed Helm release left outside Terraform state.

The learning history is kept separately in the repository-level docs/sessions/ directory. Session notes explain why the current code and run order look the way they do; they are not Terraform configuration and should not be mixed into an Acme root.

Why 5 separate Terraform repos instead of one

This is the distinctive architectural choice worth understanding on its own — it's a distributed-systems pattern applied to infrastructure code, not just an organizational preference. Each repo:

  • Has its own lifecycle — the database shouldn't be destroyed/recreated just because the backend redeploys, and the frontend shouldn't need a plan run every time a DB migration ships.
  • Has its own state file, with no shared backend config — blast radius of a bad apply is contained to one repo. The real project does this with backend "gcs" {} per repo; this copy uses backend "local" per repo so it needs no pre-existing bucket. Same isolation property either way: what matters is one state file per root, not where that file lives.
  • Publishes what other repos need, and nothing else, through Consul rather than terraform_remote_state. terraform_remote_state would create a hard coupling to another repo's entire state file (including things it never meant to expose); a Consul KV write is a deliberate, versioned, minimal publish/subscribe contract instead — closer to a service publishing an API contract than one service reading another's database directly.

Dependency graph

        ┌────────────────────────┐
        │    sample-program/      │  publishes: GKE clusters (p + np), VPC,
        │                        │              public DNS zone
        └────────────┬───────────┘
                      │
        ┌─────────────┼─────────────┐
        ▼                            ▼
┌──────────────┐            ┌──────────────┐
│  cloudsql/    │            │              │
│               │            │              │
│  publishes:   │            │              │
│  connection   │            │              │
│  info, grant/ │            │              │
│  revoke URLs  │            │              │
└───────┬───────┘            │              │
        │                    │              │
        ▼                    ▼              │
┌───────────────────────────────┐           │
│           backend/             │◄──────────┘
│                                │   reads: sso_secret_id (grants itself
│  publishes: backend_service_url│   secretAccessor), cloudsql outputs
│  service_account.email         │
└───────────────┬────────────────┘
                 │
                 ▼
        ┌─────────────────┐
        │   frontend/       │  reads: backend_service_url, sso_client_id
        │                   │  (never touches sso_secret_id/client_secret)
        └───────────────────┘

infrastructure also reads backend's published service-account email (infrastructure/infra/iam.tf) — the grant runs in infrastructure (it owns the secret) even though the dependency on knowing who to grant flows the other way. This is worth sitting with: "who owns the resource" and "who initiates the Terraform read" aren't always the same repo.

Consul key map (the actual contract between repos)

Path Written by Read by
.../sample-program/default sample-program/infra backend, frontend
.../infrastructure/{dev,qa,default} infrastructure/infra backend, frontend
.../cloudsql/{dev,qa,default} cloudsql/infra backend
.../backend/{workspace} backend/infra frontend, infrastructure

Every one of these paths is decoded with jsondecode(...) on the reading side and encoded with jsonencode(...)/consul_keys on the writing side — if you ever change a field name on one side, grep the whole sandbox for it before assuming the change is safe. That's the tradeoff of a JSON-over-KV contract instead of typed Terraform module outputs: nothing catches a typo at plan time.

What to trace first

  1. sample-program/infra/main.tf — the for_each-over-modules pattern creating zero, one, or two GKE clusters depending on the enable flags, then the consul_keys publish at the bottom that builds the exact JSON shape every other repo's local.program_gcp expects.
  2. infrastructure/infra/main.tf and secrets.tf — good warm-up for the for_each-over-environments pattern (note: this repo does not use terraform.workspace per environment the way the other repos do — it runs a single workspace and fans out with for_each instead. Ask yourself why that choice might make sense here specifically.)
  3. cloudsql/infra/ — a self-contained, single-purpose repo. Good for seeing a full main.tf → iam.tf → outputs.tf flow without the added complexity backend has.
  4. backend/infra/main.tf locals, then cloudsql.tf, then the depends_on block at the bottom of main.tf — still the most architecturally dense file in the sandbox.
  5. frontend/infra/vip.tf + iam.tf — contrast against backend's iam.tf: notice everything backend needs (SQL roles, shell_script bridge, sso_secret_id access) that frontend deliberately does not need, and why (no database, and sso_client_id is non-sensitive so it skips Secret Manager IAM entirely).