Skip to content

Sandbox runbook

Single-sourced

This page renders acme-sampleapp-multirepo-sandbox-elaborate/RUNBOOK.md directly, so it can never drift from the scripts it documents. Edit the file in the sandbox, not here.

Everything here is runnable against a real GCP project. It is not runnable against your organization's real GKE cluster/Consul cluster — it needs its own, which this runbook sets up (a lab-sized GKE cluster you create and destroy yourself, and a local Consul dev agent standing in for a real Consul cluster).

Read the cost section before running terraform apply anywhere near the GKE modules. Everything else in this sandbox (VPC, DNS zone, Secret Manager, Cloud SQL at the smallest tier) is cheap-to-free; GKE nodes are the one thing here that meaningfully costs money per hour they're running.

0. One-time tool setup

# Auth — this sandbox uses your own gcloud credentials via Application
# Default Credentials, the same way `google_client_config.default` (used
# throughout for the kubernetes/helm provider tokens) expects.
gcloud auth application-default login
gcloud config set project <your-real-project-id>

# Local Consul dev agent — stands in for your org's real Consul cluster.
# `-dev` mode is in-memory, single-node, no ACLs, no persistence. Perfect for
# this and nothing else — never run -dev mode anywhere but your own machine.
consul agent -dev &
export CONSUL_HTTP_ADDR=http://127.0.0.1:8500

# Verify:
consul kv put test/hello world
consul kv get test/hello   # should print "world"

If you don't have the consul CLI: brew install consul (macOS) or grab a binary from releases.hashicorp.com/consul. No Docker required for -dev mode. The restore script checks for Consul before running Terraform and waits until the local agent responds; if startup fails, inspect /tmp/consul-dev.log.

1. Point the tfvars at your own project

All five roots already ship a committed terraform.tfvars holding lab placeholders, so there is nothing to copy — edit them in place. At minimum, sample-program/infra/terraform.tfvars needs your real project_id; every other value works as shipped.

grep -rn "my-devops-journey-502420" */infra/terraform.tfvars

Each terraform.tfvars.example sits alongside as the annotated reference copy, explaining what every variable does and which ones cost money. Read the example, edit the real file. To reset a root back to shipped defaults:

cp sample-program/infra/terraform.tfvars.example sample-program/infra/terraform.tfvars

2. Resume or bootstrap the sandbox

For normal learning sessions, run the restore script from the Acme directory:

./restore-session.sh

The script resolves its own directory, preserves the local Consul KV data, runs terraform plan -detailed-exitcode, and applies only when a change is needed. A successful destroy-all.sh records a local clean-destroy marker, so the next plain restore-session.sh can rebuild the normal learning environment without an extra flag. Unexpectedly empty or missing state still fails closed because it may indicate lost ownership; use --bootstrap only for the first intentional setup after confirming no matching live resources exist. If the local Consul dev agent was restarted, its in-memory data is gone; republish tracked outputs explicitly:

./restore-session.sh --refresh-consul

Before applying the backend, the script verifies that the GKE cluster has at least one Ready node and waits up to five minutes for startup. This avoids a Helm timeout when the cluster has no usable node pool. The development values request only 100m CPU and 256Mi memory so the backend can schedule on the single non-production lab node. After a clean destroy removes Artifact Registry, the script rebuilds backend/app for linux/amd64 and pushes the configured image automatically; Docker Desktop must be running when that image is absent.

If a previous Helm attempt left a failed release outside Terraform state, repair it explicitly:

./restore-session.sh --repair-failed-helm

Use --skip-backend when you intentionally want to restore only the platform and shared infrastructure:

./restore-session.sh --skip-backend

For the first intentional setup only, allow creation from empty state:

./restore-session.sh --bootstrap

Do not use --bootstrap as a recovery shortcut for a missing state file. Recover or import the existing state first; otherwise Terraform cannot know that the real resources already belong to this configuration.

3. Apply order (matches the dependency graph in README.md)

Each repo is terraform init && terraform plan && terraform apply from its own infra/ directory — they are genuinely independent state files, run independently, exactly like the real 5-repo project.

cd sample-program/infra
terraform init
terraform plan     # safe — enable_gke_p/np default to false, nothing costs money yet

Stop here and read the cost section below before going further.

# When ready to actually stand up a cluster (recommended: np only, it's cheaper
# and it's what dev/qa/review environments use):
#   edit terraform.tfvars: enable_gke_np = true
terraform apply

cd ../../infrastructure/infra
terraform init && terraform apply

cd ../../cloudsql/infra
terraform init && terraform apply

cd ../../backend/infra
terraform init && terraform apply    # requires kubectl-style access — this repo's
                                       # kubernetes/helm providers authenticate using
                                       # the GKE cluster sample-program just published

cd ../../frontend/infra
terraform init && terraform apply

If any apply fails with something like "no host found" or a provider config error, it almost always means the upstream repo hasn't been applied yet, or enable_gke_np/enable_gke_p is still false — the dependency graph in README.md is the actual troubleshooting order.

4. Verifying a learning session

Use the read-only health check after restoring or before teaching a session:

./verify-session.sh

It checks local tools and ADC, Consul contracts, Terraform state files, the GKE cluster and node pool, Kubernetes connectivity, Ready nodes, the backend namespace/workloads, recent events, and Helm status. It exits nonzero when a required check fails. Add Terraform refresh plans when you want drift/change checks as well:

./verify-session.sh --plans

--plans is read-only but can take longer because each Terraform root refreshes its providers and compares state with the live resources.

5. Tearing down

The destroy helper resolves its own directory, so it can be run from any working directory. It asks you to type destroy once, then creates and applies saved destroy plans in reverse dependency order:

./destroy-all.sh

It checks every existing frontend, backend, and Cloud SQL workspace, skips empty state, destroys in reverse dependency order, clears stale local Consul contracts, and records the clean teardown used by the next plain restore. Both lifecycle scripts print potentially billable versus no-direct-charge Terraform resource action counts; these are resource counts, not price estimates. A failed Helm release that is not in Terraform state is removed when its GKE cluster is destroyed; if you are preserving the cluster and only repairing the backend, use restore-session.sh --repair-failed-helm instead.

Reverse order, same as any dependency graph:

cd frontend/infra    && terraform destroy
cd ../../backend/infra      && terraform destroy
cd ../../cloudsql/infra     && terraform destroy
cd ../../infrastructure/infra && terraform destroy
cd ../../sample-program/infra && terraform destroy

Kill the Consul dev agent when done: kill %1 (or find it with jobs). -dev mode is in-memory anyway — nothing persists once it's killed. 

kill $(pgrep consul)

Cost warning — read before enabling GKE

enable_gke_p / enable_gke_np default to false specifically so you can plan and read the whole graph without spending anything. Once you flip either to true:

  • You're paying for 1 running e2-small node (spot pricing for np, on-demand for p) — order of cents/hour, but it accrues the whole time it's running, not just while you're actively using it.
  • Turn it off when you're done for the session. terraform destroy from sample-program/infra removes the cluster and node pool. There's no "pause" — destroy and re-apply next session is the normal workflow for a lab cluster.
  • Consider enabling only enable_gke_np rather than both — one cluster is enough to exercise the entire dependency graph end to end (backend and frontend both read local.program_gcp keyed by p_or_np, and any non-default Terraform workspace resolves to np).
  • Check current GKE/Compute Engine pricing for your region before leaving anything running unattended — prices change and this file won't stay current.
sh restore-session.sh --repair-failed-helm
sh verify-session.sh