platformNeil Godfreyapplied

Foundation layer of a Kubernetes homelab

The foundation layer of a three-cluster homelab on one Mac: Lima runs the Linux VMs, K3s is the Kubernetes distribution in each VM, Rancher manages all three clusters (with Fleet delivering its agents), Traefik is the ingress and cert-manager issues the in-cluster certificates. All of it is open source and free. Every other layer, from delivery to backup, runs on top of these six pieces.

Updated October 17, 2026

The foundation layer is the base of a three-cluster Kubernetes homelab that runs on one Mac. It turns the Mac into three Linux VMs, puts a single-server K3s cluster in each one (management, beta and prod), gives the operator one place to manage all three, and supplies the ingress and certificates that every workload needs. Delivery, observability, security, data, backup and operations all sit on top of it.

The services in this layer

Foundation, by clusterHover, tap or tab to a service. The red lines show what it talks to.

One Mac, three Linux VMs

FoundationThe VMs, Kubernetes and the control plane that everything else runs on.

All three clusters

Management

Lima: Runs the three Linux VMs on one Mac with Apple’s Virtualization framework. Each VM is a whole cluster.

K3s: Lightweight Kubernetes in every VM, with a SQLite datastore and a built-in Helm controller.

Rancher: One UI and API over all three clusters. Sign-in goes through Keycloak; a break-glass admin covers the rest.

Fleet: Rancher’s GitOps engine. Here it only delivers Rancher’s own agents to the clusters; Argo CD does the rest.

Traefik: The ingress controller in each cluster. Routes incoming HTTP to the right service.

cert-manager: Issues and renews the TLS certificates used inside the clusters.

Argo CD: The GitOps hub. Keeps beta in step with the beta branch and prod with main; prod syncs only when started by hand.

Argo Workflows: Runs the image build pipelines, one workflow per product image.

BuildKit: Builds the container images inside the workflows, with layer caching.

Registry: The private image registry (CNCF Distribution) that every cluster pulls from.

Prometheus: The metrics hub on Management. Beta and prod run Prometheus agents that write to it.

Alertmanager: Groups, silences and routes alerts to email, including failed backup jobs.

Grafana: Dashboards over the metrics, logs and traces, behind Keycloak sign-in.

Loki: The central log store for every cluster.

Grafana Alloy: The collector on every cluster that ships logs to Loki.

Tempo: The trace store, next to Loki.

OpenTelemetry Operator: Manages OpenTelemetry collectors and auto-instrumentation, trialled on beta first.

Pyrra: Turns SLO definitions into Prometheus rules and burn-rate alerts.

Blackbox Exporter: Uptime probes against the public sites and APIs, from outside the app.

OpenCost: Shows what each namespace and workload costs in CPU and memory.

Keycloak: Single sign-on for the products, Rancher, Grafana and kubectl.

OpenBao: The secrets store, an open-source fork of Vault. The source of truth for app secrets.

External Secrets: Copies secrets from OpenBao into Kubernetes Secrets on beta and prod.

Kyverno: The policy engine: admission rules on what workloads may do.

Falco: Runtime threat detection from kernel events; findings arrive as alerts.

Trivy Operator: Scans the running images and their configuration for known vulnerabilities.

Cloudflare Tunnel: Publishes the sites with no open inbound ports. Cloudflare Access gates the admin hosts.

CloudNativePG: The Postgres operator: failover, nightly base backups and continuous WAL archiving to the backup store.

Postgres: The product databases, one per product, with business logic in stored procedures.

Neo4j: The knowledge graphs behind the agents. Dumped nightly to the backup store.

Redis: Cache and queues for the apps and for Airflow.

MinIO: S3-compatible object storage for the apps’ files.

SeaweedFS: The S3 backup store every cluster writes to. Its data sits in a folder on the Mac, so backups outlive a lost VM.

Velero: Nightly backups of every cluster’s objects and volumes, with Kopia file copies.

Rancher Backup: An encrypted nightly backup of Rancher itself. Drilled on a throwaway VM: back in under a minute.

Apache Airflow: One shared scheduler with a worker per product for the data pipelines.

Backstage: The developer portal: a catalog of every service with its docs.

Reloader: Restarts a workload when its ConfigMap or Secret changes, so new config takes effect.

System Upgrade Controller: Carries out K3s upgrades started from Rancher, one cluster at a time.

Lima 2.2.1 is the VM layer. It uses Apple's Virtualization framework (vz) to run three Ubuntu 26.04 VMs: management (8 CPU, 24 GiB), beta (6 CPU, 16 GiB) and prod (8 CPU, 36 GiB). launchd starts them at login. The VM templates are code, with one shared base file and one small file per VM, and each template forwards only the ports it lists explicitly.

K3s v1.36.5+k3s1 is the Kubernetes distribution on all three VMs. Each cluster is a single server on the default SQLite datastore through Kine. The built-in Helm controller installs the platform charts from HelmChart objects, and the bundled Traefik ingress and local-path storage stay on. A nightly snapshot copies the datastore together with its token, credential and TLS files to the backup store, with 14 days kept.

Rancher v2.15.2 is the management plane. It runs as one replica on the management cluster and manages the beta and prod clusters as imported clusters, whose agents dial out to it. Sign-in goes through Keycloak over OIDC, restricted to one user, and Rancher drives K3s upgrades through the System Upgrade Controller.

Fleet v0.16.2 comes installed with Rancher. Rancher's continuous-delivery feature is off and there are no GitRepos, so Fleet has one job: delivering Rancher's own agent bundles to the clusters. Argo CD does all application GitOps.

Traefik v3.7.13 is the ingress on every cluster, as K3s bundles it, with its settings in a HelmChartConfig. It serves Rancher's ingress and the ingresses for products such as paiddaily.io and sagedaily.io.

cert-manager v1.21.2 runs on every cluster and issues in-cluster certificates: Rancher's self-signed CA flow, the TLS certificates for CloudNativePG Postgres clusters, and webhook serving certificates.

How the pieces fit

The layer has three flows. A cluster comes up from a VM template and gets its ingress from the K3s Helm controller. A public request reaches a product through Cloudflare and Traefik, and cert-manager covers the hops inside the cluster. An upgrade starts in Rancher and moves through the clusters one at a time.

How the foundation layer runs a cluster, a request and an upgrade

Step 1 of 5. launchd starts the VM at login. On a fresh VM, the template's provision script installs K3s at a pinned version, so a rebuilt VM comes back on the same release.

Three single-server K3s clusters in Lima VMs on one Mac, managed from Rancher on the management cluster.

Two details hold the flows together. The management cluster reaches beta and prod over Lima's user-v2 network, where the VMs see each other by name, and the Mac reaches each VM over vzNAT. Only explicitly forwarded ports reach the Mac, and a new forward needs a VM restart because the host agent reads the rules only at start.

Choices and trade-offs

Choice	What runs	Why	What it costs
VM layer	Lima with `vz`	Free, open-source Linux VMs on a Mac, with templates kept as code	Port forwards need a VM restart to change
Kubernetes distribution	K3s, single server per cluster	One binary with ingress, storage and a Helm controller included	No control-plane HA; the nightly snapshot is the recovery path
Datastore	SQLite through Kine	Matches the one-server shape of each cluster	Moving to HA means embedded etcd and a conversion
Cluster management	Rancher, one replica	One sign-in and one place to run upgrades for three clusters	If Rancher is down, workloads run but the UI and upgrades stop
GitOps	Argo CD for apps, Fleet for Rancher agents only	One controller owns each application, so two tools never fight over state	Fleet stays installed but unused for apps
Ingress	Bundled Traefik, tuned with HelmChartConfig	Kept as K3s ships it, so it survives K3s upgrades	Traefik patches wait for a K3s release
Public TLS	Cloudflare edge, through a tunnel	No open ports and no ACME credentials in a cluster	Public traffic depends on Cloudflare
In-cluster TLS	cert-manager	Standard issuer for database, webhook and Rancher certificates	One more controller per cluster

Each cluster uses a single K3s server because the three clusters already split the risk by environment: beta takes upgrades first, and prod follows. The cost is that a failed server means a restore, which is why the K3s snapshot copies the token with the datastore. A datastore copy without its token is not usable.

Rancher's continuous-delivery feature was turned off because Argo CD already does application GitOps. With the feature off, Fleet stays installed, because Rancher does not let it be removed fully, and it keeps delivering the agent bundles.

Traefik was kept instead of being replaced with another ingress controller. Its configuration lives in a HelmChartConfig, so the packaged chart stays as K3s ships it. On the management cluster, the websecure port is 8444 because ports 443 and 8443 on the Mac were already taken. At the time of writing the running Traefik is two patches behind v3.7.15, and both newer patches fix security advisories, so the next K3s upgrade carries a security fix.

cert-manager has no ACME issuer, because public TLS terminates at Cloudflare. That keeps its scope to private certificates whose clients are other services in the same stack, and it keeps public DNS credentials out of the clusters.

Rancher itself is protected twice over. The public Rancher URL sits behind Cloudflare Access, and the local admin login stays as a break-glass account with its password in OpenBao. A nightly encrypted Rancher Backup covers Rancher's resources, and a restore drill on a throwaway VM brought them back in 51 seconds.

The services in this layer

How the pieces fit

Choices and trade-offs

Related