Setting up a homelab from scratch: backups and restore - Homelab (06/06)
A backup you have never restored is a hope, not a backup. This post explains how the homelab is backed up — what, how and where to — and the part almost nobody talks about: how it is restored, and what I know is still missing.
You already know rule number one from previous posts: everything converges on spinoza, the TrueNAS box on the Atenas network that runs MinIO, our S3-compatible object store. A single offsite destination, on the other side of the WireGuard tunnel, far from the Roma clusters. And everything that gets backed up reaches the MinIO store on spinoza through cervantes, the Tailscale gateway.
What gets backed up and how
PostgreSQL: physical and logical
The database is the most valuable thing in the homelab, so it has two independent layers:
- Physical, with CNPG and barman: the CloudNativePG operator archives the WAL (Write-Ahead Log, the record of every change to the database) continuously and takes a daily base backup at 02:15, with 30 days of retention. With WAL + base backup you get PITR (Point-In-Time Recovery): I can restore the database to any point within those 30 days, not just to last night's backup.
- Logical, with a daily dump: in addition, a pre-backup pod runs
pg_dumpin custom format on the critical databases. It is defense in depth — if the physical format fails, the logical dump is portable to any PostgreSQL—.
Typesense: file-level or reindex
The search index is backed up with K8up at the file level over the PVC, every day at 04:00. The schedule carries a 2 GiB memory limit because restic was dying from OOM without it — a detail learned the hard way—. And if everything else fails, the index can always be regenerated from Postgres: it is a derivative, not a source.
What is NOT backed up: Redis
Redis has no backup. It is a queue broker — its content is transient by design—. If it is lost, it is recreated and the jobs are requeued. Backing up the ephemeral only costs money.
And on escipion, ZFS snapshots
The escipion node adds its own layer of protection: its data volumes live in a ZFS pool (OpenEBS ZFS LocalPV). A scheduler keeps local snapshots retained and VolSync replicates them to restic — Harbor and the monitoring PVCs are covered twice: local snapshot for instant rollback and offsite replica for the real disaster—.
The rest of the fleet: tolstoi
The Docker hosts and the VPSs run tolstoi, the Restic backup agent we talked about in the security post: a daily copy at 03:00 with retention of 7 daily, 4 weekly, 12 monthly and 7 yearly backups, reaching MinIO through the Tailscale gateway. In the clusters, K8up plays the same role per namespace.
How a restore works
The restore runbook has an order of preference:
- CNPG recovery from the barman archive — the preferred path: the operator boots a new cluster directly from the physical copies.
- Logical dump — if only the application and auth databases are needed, the
pg_dumpis restored by hand. - Typesense — from the K8up snapshot, or by reindexing from Postgres if the snapshot fails.
The outstanding debt
Two things that are not solved and are worth saying out loud. First: there are no automated restore drills — restores have been done by hand in tests, but no process verifies every week that the backups actually boot—. Second: MinIO is a single point — spinoza itself does not have its protection represented in Git; if Atenas burns down, so does the offsite—.
Neither of the two invalidates the system, but they are next on the list.
Closing
A decent backup system is boring on the inside: single destination, two layers for what is critical, staggered retention and a written runbook. What is interesting is what it enables: rebuilding any piece of the homelab without panic.
Previous: Post 5 - Observability
The full series
Building a homelab from scratch: observability - Homelab (05/06)
Building a homelab from scratch: security and operations - Homelab (04/06)
Building a homelab from scratch: networking - Homelab (02/06)
Postiz: scheduling social media posts from your own server
Building a homelab from scratch: Kubernetes and GitOps - Homelab (03/06)
