Build Daily

Tinley Park · October 29, 2026
platformNeil Godfreyapplied

Backup and recovery layer of a Kubernetes homelab

The backup and recovery layer of a three-cluster K3s homelab managed by Rancher. Velero copies each cluster's objects and volumes, Rancher Backup saves Rancher itself, and K3s snapshots, CloudNativePG base backups with WAL, Neo4j dumps, pg_dump and OpenBao Raft snapshots cover the rest. Every job writes to one SeaweedFS S3 store whose data sits on the Mac host, and a nightly retention job checks that each set is fresh. All of it is open source and free.

Updated October 17, 2026

The backup and recovery layer makes sure that a lost namespace, database or whole VM can be rebuilt from copies taken the night before. In this homelab, three K3s clusters (management, beta and prod) run as Lima VMs on one Mac, and Rancher manages all three. Each kind of state has its own backup job, and every job writes to one S3 store on the management cluster whose data lives outside the VMs.

The services in this layer

Backup and recovery, by clusterHover, tap or tab to a service. The red lines show what it talks to.
One Mac, three Linux VMs
Backup and recoveryGetting everything back after a bad day, and proving it with drills.
All three clusters
Management

Velero v1.18.4 runs on all three clusters with the AWS plugin and Kopia file-system backup. One schedule per cluster runs at 02:00 Chicago time, keeps 14 days, and covers every namespace except the one that holds the backup store. A volume policy skips the CloudNativePG volumes, Prometheus data, caches and emptyDirs. Each cluster writes to its own prefix in the velero bucket, and two alerts, VeleroBackupFailed and VeleroBackupTooOld, watch the results.

Rancher Backup v11.0.3 runs on the management cluster and takes an encrypted nightly backup of Rancher's own resources: users, auth settings and cluster registrations. The operator only speaks HTTPS S3, so it writes to a PVC, and a CronJob copies each new file to the store. A restore drill on a throwaway VM took 51 seconds, and Rancher came back with all users, auth and clusters.

The rest of the layer is jobs that belong to other services:

  • K3s keeps its datastore in SQLite. A nightly job runs sqlite3 .backup and copies the token, credentials and TLS files to the store, kept 14 days.
  • CloudNativePG takes a nightly base backup and streams WAL continuously through the Barman Cloud plugin, kept 14 days. Point-in-time restore has been drilled.
  • Postgres also gets a nightly pg_dump per database: 7 days on the VM, 14 in the store.
  • Neo4j Community has no online backup, so CronJobs stop each graph for 20 to 60 seconds, dump it, restart it and upload the dump, kept 14 days.
  • OpenBao sends a Raft snapshot to the store every night.

SeaweedFS 4.48 is that store. It runs as one weed server -s3 process on the management cluster, with the buckets postgres, neo4j, velero and etcd and one S3 identity per cluster limited to its own buckets. Its data directory is a Mac folder mounted into the VM, so the backups survive a lost VM. A nightly retention job fails when the newest copy in a set is older than 36 hours, and keeps one copy per month of the small sets for six months.

How the pieces fit

Two flows cross this layer. Each night, the jobs above write into the store and the retention job checks them. After a VM is lost, the rebuild reads them back in a fixed order.

How backups land in the store and come back

Step 1 of 6. A job copies the SQLite datastore with sqlite3 .backup, so the copy is consistent while K3s runs. The token, credentials and TLS files go with it.

Every job writes to one SeaweedFS store whose data sits on the Mac host, outside the VMs.

The store is the one place every flow meets, which is why its data is kept outside the VMs. If the management VM itself is lost, SeaweedFS starts again on the same Mac folder before anything else is restored. Each tool restores only its own kind of state: Velero does not know about Rancher's encryption or Postgres WAL, and CloudNativePG does not know about Kubernetes objects. The pg_dump and Neo4j files are the slower path back for a single database or graph.

Choices and trade-offs

  • SeaweedFS for the store. It replaced MinIO as the backup target because MinIO's community edition is archived. MinIO still serves app files on prod, but backups go to a store that gets fixes. One process on one node keeps a single copy, so the Mac folder is the thing to protect.
  • Data outside the VMs. A VM can be rebuilt from its template. A backup inside that VM would be lost with it, so the store's directory is mounted from the host.
  • One identity per cluster. Each cluster can write only to its own buckets. A compromised cluster cannot read or delete another cluster's backups.
  • Velero skips what has its own backup. Postgres gets a consistent base backup with WAL from CloudNativePG, so Velero does not take file copies of those volumes. Prometheus data and caches can be rebuilt or scraped again, so they are skipped too.
  • Two Postgres backups. CloudNativePG covers point-in-time restore of the whole cluster. The per-database pg_dump gives a small, portable file for one product's data.
  • Downtime for Neo4j. Community edition has no online backup. A stop of 20 to 60 seconds per graph each night was chosen over a paid edition.
  • Copy, never sync, for Rancher Backup. The chart names its PVC after the Helm release revision, so an upgrade starts a new, empty volume. A sync from that volume would delete every backup in the bucket.
  • Freshness as the alarm. A job that stops running produces no error. The 36-hour check and VeleroBackupTooOld catch the silence.
  • Drills, not assumptions. The Rancher restore and the Postgres point-in-time restore have both been run end to end, and the Rancher drill found the PVC naming issue.

Related