platformNeil Godfreyapplied

Delivery layer of a Kubernetes homelab

The delivery layer of a three-cluster K3s and Rancher homelab turns a merged commit into a running workload. Argo Workflows runs one build per product image, BuildKit builds it with a kept layer cache, a CNCF Distribution registry stores it under its git SHA, and Argo CD syncs the manifests from Git to the beta and prod clusters by hand. All four tools are Apache 2.0, free and run on the management cluster.

Updated October 17, 2026

The delivery layer is the part of the homelab that moves code from a Git branch to a running pod. The platform is three K3s clusters, run as Lima VMs on one Mac and managed by Rancher: a management cluster, a beta cluster and a prod cluster. Every delivery service runs on the management cluster. The beta and prod clusters only pull images and receive manifests, so they carry no build tools and no CI credentials.

The services in this layer

Delivery, by clusterHover, tap or tab to a service. The red lines show what it talks to.

One Mac, three Linux VMs

DeliveryHow code becomes images, and images become running workloads.

Management

Lima: Runs the three Linux VMs on one Mac with Apple’s Virtualization framework. Each VM is a whole cluster.

K3s: Lightweight Kubernetes in every VM, with a SQLite datastore and a built-in Helm controller.

Rancher: One UI and API over all three clusters. Sign-in goes through Keycloak; a break-glass admin covers the rest.

Fleet: Rancher’s GitOps engine. Here it only delivers Rancher’s own agents to the clusters; Argo CD does the rest.

Traefik: The ingress controller in each cluster. Routes incoming HTTP to the right service.

cert-manager: Issues and renews the TLS certificates used inside the clusters.

Argo CD: The GitOps hub. Keeps beta in step with the beta branch and prod with main; prod syncs only when started by hand.

Argo Workflows: Runs the image build pipelines, one workflow per product image.

BuildKit: Builds the container images inside the workflows, with layer caching.

Registry: The private image registry (CNCF Distribution) that every cluster pulls from.

Prometheus: The metrics hub on Management. Beta and prod run Prometheus agents that write to it.

Alertmanager: Groups, silences and routes alerts to email, including failed backup jobs.

Grafana: Dashboards over the metrics, logs and traces, behind Keycloak sign-in.

Loki: The central log store for every cluster.

Grafana Alloy: The collector on every cluster that ships logs to Loki.

Tempo: The trace store, next to Loki.

OpenTelemetry Operator: Manages OpenTelemetry collectors and auto-instrumentation, trialled on beta first.

Pyrra: Turns SLO definitions into Prometheus rules and burn-rate alerts.

Blackbox Exporter: Uptime probes against the public sites and APIs, from outside the app.

OpenCost: Shows what each namespace and workload costs in CPU and memory.

Keycloak: Single sign-on for the products, Rancher, Grafana and kubectl.

OpenBao: The secrets store, an open-source fork of Vault. The source of truth for app secrets.

External Secrets: Copies secrets from OpenBao into Kubernetes Secrets on beta and prod.

Kyverno: The policy engine: admission rules on what workloads may do.

Falco: Runtime threat detection from kernel events; findings arrive as alerts.

Trivy Operator: Scans the running images and their configuration for known vulnerabilities.

Cloudflare Tunnel: Publishes the sites with no open inbound ports. Cloudflare Access gates the admin hosts.

CloudNativePG: The Postgres operator: failover, nightly base backups and continuous WAL archiving to the backup store.

Postgres: The product databases, one per product, with business logic in stored procedures.

Neo4j: The knowledge graphs behind the agents. Dumped nightly to the backup store.

Redis: Cache and queues for the apps and for Airflow.

MinIO: S3-compatible object storage for the apps’ files.

SeaweedFS: The S3 backup store every cluster writes to. Its data sits in a folder on the Mac, so backups outlive a lost VM.

Velero: Nightly backups of every cluster’s objects and volumes, with Kopia file copies.

Rancher Backup: An encrypted nightly backup of Rancher itself. Drilled on a throwaway VM: back in under a minute.

Apache Airflow: One shared scheduler with a worker per product for the data pipelines.

Backstage: The developer portal: a catalog of every service with its docs.

Reloader: Restarts a workload when its ConfigMap or Secret changes, so new config takes effect.

System Upgrade Controller: Carries out K3s upgrades started from Rancher, one cluster at a time.

Argo Workflows v4.1.5 is the build system. Each product image has its own workflow, which runs as pods on the management cluster. The Argo UI is published on the Mac, where a failed build can be read step by step and retried.

BuildKit v0.34.0 does the image builds. The workflow templates hand each build to a buildkitd daemon. The daemon keeps its layer cache between runs, so a small code change rebuilds only the layers above it and reuses the dependency layers.

Distribution 3.1.2, the CNCF registry server, stores every image the platform builds. All three clusters reach it on port 5001, because macOS holds port 5000 for the AirPlay Receiver. Traffic is plain HTTP inside the platform's private network. Every image is tagged with the git SHA of the commit it came from, so one tag names one build.

Argo CD v3.5.5 is the GitOps controller. It runs as a hub on the management cluster with two Applications. The beta Application tracks the beta branch and the prod Application tracks main. Neither one syncs automatically. Sign-in goes through Keycloak by way of the bundled Dex, and Argo CD reads the repository with a read-only deploy key.

How the pieces fit

The layer has two flows that meet at the registry. The build flow turns a commit into an image. The deploy flow makes a cluster match the manifests in Git, and those manifests name the image tags.

From a merged commit to a running workload

Step 1 of 3. The merged commit is the input to the workflow for that product image. The run shows in the Argo UI, where it can be watched or retried.

Build and deploy are separate steps. Argo CD changes nothing until a sync is started by hand.

Because the tag is a git SHA, the image that runs on prod is the same image that ran on beta. The promotion moves a reference in Git; it does not trigger a rebuild. The registry sits between the two flows, so a build can finish while no deploy is waiting, and a revert in Git can point a cluster back at an older tag that is still in the registry.

Choices and trade-offs

Builds moved onto the cluster. Images used to be built locally in Docker Desktop. Docker Desktop is now retired, and builds run on cluster compute next to the registry. The Argo UI gives each build a record with per-step logs, which local builds did not have. The cost is more to maintain: a workflow controller, a build daemon and a registry, where before there was one desktop app.

Argo Workflows for CI. The builds are Kubernetes resources that run on the same cluster as the registry and the GitOps controller. Each build has direct access to a registry that is only on the private network. Argo Workflows and Argo CD both belong to the CNCF Argo project.

BuildKit as a daemon. BuildKit builds images without a Docker daemon in the pod. A long-running buildkitd keeps its layer cache between runs, which matters more than build isolation on a single Mac with limited CPU.

A plain registry. Distribution is a single binary that implements the OCI Distribution Spec. It has no UI, no scanning and no replication, and none of those is needed here. Vulnerability scanning is done by Trivy Operator on the running workloads on beta, and Kyverno policies on beta and prod include image registry rules. Without TLS or registry auth, the registry is safe only while it stays on the private network. BuildKit and each cluster's container runtime must be configured to treat it as an HTTP endpoint.

Argo CD for all app GitOps, synced by hand. Rancher installs Fleet, but Fleet's continuous-delivery feature is off and it delivers only Rancher's own agent bundles. Argo CD is the only controller for applications, so no workload has two controllers that could disagree about its state. Automated sync was ruled out for both clusters. A merge to main marks prod OutOfSync and waits, so a person reads the diff before prod changes. The trade-off is that every release needs one manual step, including releases to beta.

Separate build and deploy. Argo Workflows only builds and pushes. Argo CD only applies what is in Git. A broken build cannot change a running cluster, and a sync never depends on a build that is still running.

Foundation layer: Lima, K3s, Rancher, Fleet, Traefik and cert-manager.
Observability layer
Security layer
Data layer
Backup layer
Operations layer
The stack map post
Argo CD documentation
Argo Workflows documentation
BuildKit documentation
CNCF Distribution documentation

The services in this layer

How the pieces fit

Choices and trade-offs

Related