Lab 16 · Disaster Recovery: Backup & Restore with Velero¶
Core Lesson: Every lab rebuilds forward (lab 10 cold rebuild). DR assumes you can't: etcd corrupts, a namespace gets kubectl deleted, a region fails. Velero snapshots cluster resources (+ PV data) to object storage and restores them — and the drill that matters is the restore, not the backup.
GCP Twin: Velero on GKE with a GCS bucket + Workload Identity is the standard pattern; Cloud SQL has its own PITR — know which tool owns which layer.
1. Architecture Decision: What Does "Backup" Even Mean?¶
Layer
Owned by
Restores
Cluster resources (deployments, secrets, RBAC)
Velero (or GitOps repo!)
deleted objects, config drift
PV data
Velero + restic/CSI snapshots
volume contents
Managed DB data
Cloud SQL PITR / backups (NOT velero)
DB to point-in-time
Desired state
your GitOps repo
the best restore: re-apply everything
The senior-engineer answer: GitOps IS your resource backup —
flux reconcile rebuilds resources. Velero covers what git doesn't:
cluster-created objects, PV data, and the world between commits.
RPO (how much data can you lose?)
RTO (how fast back?)
Design consequence
hours
hours
daily velero schedule is fine
minutes
minutes
CSI snapshots + tested restore runbook
zero
minutes
multi-region + replication (lab 11) — backup alone can't get there
# minio as the local "GCS bucket"helminstallvelerovmware-tanzu/velero\-nvelero--create-namespace\--setdeployNodeAgent=true\--setconfiguration.backupStorageLocation[0].provider=aws\--setconfiguration.backupStorageLocation[0].bucket=velero\--setconfiguration.backupStorageLocation[0].config.region=minio\--setconfiguration.backupStorageLocation[0].config.s3Url=http://minio.velero.svc:9000
# (GCP: provider=gcp + bucket + WI — same object model)
Step 2: Create state worth losing, then back it up¶
kubectlcreatensdr-demo
kubectl-ndr-democreatedeploymentapp--image=nginx--replicas=2kubectl-ndr-democreateconfigmapkeepme--from-literal=data=precious
velerobackupcreatedr-demo-backup--include-namespacesdr-demo--wait
velerobackupdescribedr-demo-backup# Phase: Completed — objects + metadata in the bucket
Step 3: The drill — delete the namespace, restore it¶
kubectldeletensdr-demo# the disasterkubectlgetnsdr-demo# gonevelerorestorecreate--from-backupdr-demo-backup--wait
kubectl-ndr-demogetdeploy,cm# app + keepme are back