Build Daily

Tinley Park · October 29, 2026
platformNeil Godfreyapplied

Data layer of a Kubernetes homelab

The data layer of a three-cluster K3s and Rancher homelab: relational data, graphs, cache and queues, app files and backups. CloudNativePG runs Postgres 17 on the beta and prod clusters with one database per product, Neo4j Community holds the graphs, Redis carries cache and queue traffic, MinIO serves app files over S3, and SeaweedFS on the management cluster is the S3 backup store for every cluster. All of it is open source and self-hosted; the cost is disk, memory and the backup jobs that Community editions need.

Updated October 17, 2026

The data layer holds everything the products and agents must not lose, plus the fast, disposable state that sits in front of it. In a three-cluster K3s homelab managed by Rancher, the stores run on the beta and prod clusters, close to the apps that use them, and the backup store runs on the management cluster. Each store has one job, and each one sends its backups to the same S3 endpoint.

The services in this layer

Data, by clusterHover, tap or tab to a service. The red lines show what it talks to.
One Mac, three Linux VMs
DataThe databases and object stores the products and agents use.
Beta and Prod
Management
Prod

CloudNativePG 1.30.1 is the operator that runs Postgres on the beta and prod clusters. It manages the instance pod, its volume and the Services, and with the Barman Cloud plugin v0.15.1 it takes a nightly base backup and archives WAL continuously. A point-in-time restore has been drilled, not only configured.

Postgres 17.11 is the system of record. One cluster per environment serves every product, with one database per product, and business logic lives in stored procedures that pgTAP tests cover. Beta databases are snapshots refreshed from prod. The shared Airflow keeps its own separate Postgres.

Neo4j 5.26 Community runs on the prod cluster and holds several graphs. Community Edition allows one standard database per instance, so each graph is its own instance with its own credentials and storage.

Redis 7.4.11 runs on the prod cluster for app caching and queues. A second, separate Redis is the Celery broker for the shared Airflow. Neither instance is a system of record.

MinIO RELEASE.2025-09-07 runs on the prod cluster as S3-compatible storage for app files. Its community edition is archived upstream, and that release is the last one with published binaries. It predates the source-only release that fixes CVE-2025-62506.

SeaweedFS 4.48 runs on the management cluster as one weed server -s3 process. It is the S3 backup store for every cluster, with the buckets postgres, neo4j, velero and etcd. Its data directory is a folder on the Mac mounted into the VM, so the backups outlive a lost VM.

How the pieces fit

Two kinds of traffic cross the layer. Requests go from an app to Redis, Postgres and MinIO on the same cluster. Backups go from Postgres and Neo4j across clusters to SeaweedFS, and a restore comes back the same way.

How data and backups cross the data layer

Step 1 of 3. The API looks for a cached value first and puts queue work on Redis. A miss falls through to the database.

The stores run next to the apps on beta and prod; the backup store runs on the management cluster.

Velero also writes to SeaweedFS, in its own velero bucket, but its volume policy skips the CloudNativePG volumes. The operator's backups are the backup of record for Postgres data, so the database is not copied twice in two formats. A nightly pg_dump of each database adds a logical copy for single-database restores, kept 7 days on the VM and 14 days in the store. A backup-retention job fails when the newest copy in any set is older than 36 hours.

Choices and trade-offs

One Postgres per environment, one database per product. A server per product would multiply memory, upgrades and backup configuration on small VMs. Separate databases still give each product its own schema and roles. The cost is a shared failure domain: if that Postgres cluster goes down, every product on that environment loses its database at the same time.

An operator in place of a hand-built StatefulSet. CloudNativePG brings declared backups, WAL archiving and point-in-time restore. The trade-off is a different restart model. No Deployment or StatefulSet owns the pod, so a restart means deleting the pod and letting the operator recreate it on the same volume. kubectl rollout restart does nothing useful here.

Neo4j Community, with downtime for backups. Enterprise adds online backup and clustering, but it is a paid licence. Community keeps the graphs free at the price of a short nightly stop per graph and one instance per graph. The 5.26 LTS line gets security fixes without a feature upgrade every month.

Redis kept off the critical path. Nothing in Redis is a system of record, so it gets no backup of its own. The app cache and the Airflow broker are separate instances, so a busy cache cannot fill the broker's memory and stall pipeline runs. Both run with the default noeviction policy and no memory cap, which the broker needs. A cap with an LRU policy on the cache instance is the next setting to add.

MinIO kept running and SeaweedFS chosen for backups. MinIO's community repository is archived and gets no more fixes, and the running release is missing the CVE-2025-62506 fix. Moving the app files is a data copy plus an endpoint change, so MinIO stays in place for now and its restricted service accounts are not treated as a security boundary. The backup store had to be something with upstream maintenance, so it is SeaweedFS. That also means a MinIO failure cannot take the backups with it.

Backups off the clusters they protect. The backup store sits on the management cluster, and its data sits on the Mac's own disk. A lost prod VM does not take its backups with it. Each cluster has its own S3 identity limited to its own buckets, so a compromised cluster cannot read or delete another cluster's backups.

Related