platformNeil Godfreyapplied

Observability layer of a Kubernetes homelab

The observability layer of a three-cluster K3s and Rancher homelab: metrics, logs, traces, alerts, SLOs and cost allocation. A Prometheus hub on the management cluster receives remote-write from Prometheus agents on beta and prod, Grafana Alloy ships pod logs to one Loki, Tempo holds traces, Alertmanager sends email, Pyrra and Blackbox Exporter turn outside-in probes into SLOs, and OpenCost splits CPU and memory by product. Every tool is open source; the cost is memory on the hub cluster.

Updated October 17, 2026

The observability layer answers three questions about a three-cluster K3s homelab managed by Rancher: is something broken, where, and why. It collects metrics, logs and traces from the management, beta and prod clusters into one place, sends an email when a rule or an SLO fails, and shows how much of the hardware each product uses. Storage, rules and dashboards live on the management cluster; the beta and prod clusters run only the collectors.

The services in this layer

Observability, by clusterHover, tap or tab to a service. The red lines show what it talks to.

One Mac, three Linux VMs

ObservabilityMetrics, logs, traces, alerts, SLOs and cost, collected on the hub.

All three clusters

Beta and Prod

Management

Beta

Lima: Runs the three Linux VMs on one Mac with Apple’s Virtualization framework. Each VM is a whole cluster.

K3s: Lightweight Kubernetes in every VM, with a SQLite datastore and a built-in Helm controller.

Rancher: One UI and API over all three clusters. Sign-in goes through Keycloak; a break-glass admin covers the rest.

Fleet: Rancher’s GitOps engine. Here it only delivers Rancher’s own agents to the clusters; Argo CD does the rest.

Traefik: The ingress controller in each cluster. Routes incoming HTTP to the right service.

cert-manager: Issues and renews the TLS certificates used inside the clusters.

Argo CD: The GitOps hub. Keeps beta in step with the beta branch and prod with main; prod syncs only when started by hand.

Argo Workflows: Runs the image build pipelines, one workflow per product image.

BuildKit: Builds the container images inside the workflows, with layer caching.

Registry: The private image registry (CNCF Distribution) that every cluster pulls from.

Prometheus: The metrics hub on Management. Beta and prod run Prometheus agents that write to it.

Alertmanager: Groups, silences and routes alerts to email, including failed backup jobs.

Grafana: Dashboards over the metrics, logs and traces, behind Keycloak sign-in.

Loki: The central log store for every cluster.

Grafana Alloy: The collector on every cluster that ships logs to Loki.

Tempo: The trace store, next to Loki.

OpenTelemetry Operator: Manages OpenTelemetry collectors and auto-instrumentation, trialled on beta first.

Pyrra: Turns SLO definitions into Prometheus rules and burn-rate alerts.

Blackbox Exporter: Uptime probes against the public sites and APIs, from outside the app.

OpenCost: Shows what each namespace and workload costs in CPU and memory.

Keycloak: Single sign-on for the products, Rancher, Grafana and kubectl.

OpenBao: The secrets store, an open-source fork of Vault. The source of truth for app secrets.

External Secrets: Copies secrets from OpenBao into Kubernetes Secrets on beta and prod.

Kyverno: The policy engine: admission rules on what workloads may do.

Falco: Runtime threat detection from kernel events; findings arrive as alerts.

Trivy Operator: Scans the running images and their configuration for known vulnerabilities.

Cloudflare Tunnel: Publishes the sites with no open inbound ports. Cloudflare Access gates the admin hosts.

CloudNativePG: The Postgres operator: failover, nightly base backups and continuous WAL archiving to the backup store.

Postgres: The product databases, one per product, with business logic in stored procedures.

Neo4j: The knowledge graphs behind the agents. Dumped nightly to the backup store.

Redis: Cache and queues for the apps and for Airflow.

MinIO: S3-compatible object storage for the apps’ files.

SeaweedFS: The S3 backup store every cluster writes to. Its data sits in a folder on the Mac, so backups outlive a lost VM.

Velero: Nightly backups of every cluster’s objects and volumes, with Kopia file copies.

Rancher Backup: An encrypted nightly backup of Rancher itself. Drilled on a throwaway VM: back in under a minute.

Apache Airflow: One shared scheduler with a worker per product for the data pipelines.

Backstage: The developer portal: a catalog of every service with its docs.

Reloader: Restarts a workload when its ConfigMap or Secret changes, so new config takes effect.

System Upgrade Controller: Carries out K3s upgrades started from Rancher, one cluster at a time.

Prometheus is the metrics store. kube-prometheus-stack 92.8.0 installs a Prometheus v3.15.0 hub on the management cluster, with 15 days of retention capped at 25 GB. The beta and prod clusters run Prometheus in agent mode with node-exporter and kube-state-metrics, and remote-write every series to the hub. Every series carries a cluster label.

Alertmanager v0.34.1 receives the hub's alerts and sends them as email through Amazon SES. It groups by cluster, alert name and namespace, repeats every 12 hours, mutes Watchdog and info alerts, and gives Falco runtime events a route of their own.

Grafana 13.2.4 is the one UI for all three signals, with Prometheus, Loki and Tempo as data sources. Sign-in goes through Keycloak, and a local admin account stays as break-glass.

Loki 3.7.8 is the central log store. It runs as a single binary on the management cluster.

Grafana Alloy v1.20.1 runs as a DaemonSet on every cluster. It tails pod logs, adds the cluster label and pushes the lines to Loki.

Tempo 3.1.0 is the trace store, next to Loki on the management cluster.

OpenTelemetry Operator 0.161.0 runs on the beta cluster only, as a trial before prod. It manages the collectors and the auto-instrumentation of workloads, and sends traces to Tempo.

Pyrra v0.10.2 turns ServiceLevelObjective objects into Prometheus recording rules and multi-window burn-rate alerts on the hub.

Blackbox Exporter v0.29.0 probes the public sites and APIs over HTTPS from outside the apps. Its results feed the availability SLOs in Pyrra and the uptime alerts.

OpenCost 1.121.3 runs on beta and prod and allocates CPU and memory cost per namespace and workload from Prometheus data.

How the pieces fit

Three paths cross the layer. Logs go from a pod to Loki through Alloy. Metrics go from a spoke agent to the hub, and the hub's rules go to Alertmanager. Uptime goes from an outside probe into Pyrra's SLO math and back out through the same alert route. Grafana reads the end of every path.

How signals cross the observability layer

Step 1 of 3. The pod writes a log line to stdout. Alloy on the same node tails the container log file and adds the cluster label.

Collectors run on every cluster; storage, rules, alerts and dashboards run on the management cluster.

The cluster label is the joint that holds the layer together. Prometheus agents and Alloy both stamp it, so one Grafana dashboard, one alert rule and one log query cover all three clusters and can still be cut down to one.

Choices and trade-offs

One hub, collectors on the spokes. All three clusters are small VMs on one Mac. A full Prometheus with its own storage and rules on each cluster would triple the memory and the configuration to maintain. Agent mode keeps beta and prod to scraping. The cost is a single point of failure: when the management cluster is down, there are no dashboards and no alerts, and Alloy drops log batches after its retry window. For a homelab, one store and one query surface outweigh that gap.

Outside-in availability. In-cluster metrics cannot see an expired certificate, a broken DNS record or a down tunnel. Blackbox Exporter runs on the management cluster, not on prod, and probes the public URLs, so the prober does not fail with the cluster it watches. Pyrra builds the SLOs from those probes, so the burn-rate alerts measure what a visitor sees.

Email, tuned to be read. The only alert channel is email through Amazon SES. Grouping by cluster, alert name and namespace, a 12-hour repeat and muted info alerts keep a still-firing problem to two reminders a day. Backups need no tool of their own: the backup jobs are Kubernetes Jobs, so KubeJobFailed covers them, and a retention job fails when any backup set is older than 36 hours.

One vendor family for logs and traces. Loki and Tempo sit next to the Prometheus hub, with Grafana in front of all three. Single-binary Loki keeps the footprint to one pod, well below the volume where a scalable mode makes sense.

Tracing on trial. The OpenTelemetry Operator runs on beta only. A broken admission webhook or a bad agent image affects the beta environment, not prod. Logs and metrics do not depend on it.

Cost as a share. On owned hardware there is no cloud bill to split, so OpenCost's dollar figure is notional. The useful number is the share of CPU and memory each product takes.

Layer pages: Foundation, Delivery, Security, Data, Backup, Operations
The stack map post: A Kubernetes homelab on one Mac with Rancher, K3s and Lima
Services on the alert path from other layers: Falco, Keycloak, Velero

The services in this layer

How the pieces fit

Choices and trade-offs

Related