Operations layer of a Kubernetes homelab
The operations layer of a three-cluster K3s homelab managed by Rancher. Apache Airflow runs the products' scheduled batch pipelines, Backstage keeps the catalog of every service and its docs, Reloader rolls workloads when their ConfigMaps or Secrets change, and the System Upgrade Controller carries K3s version upgrades from Rancher to each node. All four are open source under Apache 2.0. The Airflow install is on the end-of-life 2.x line, and the move to Airflow 3 is open work.
The operations layer holds the work that keeps the platform running after the first deploy: scheduled batch jobs, the catalog of what runs where, config changes that must reach running pods, and version upgrades of Kubernetes itself. In this homelab, three K3s clusters (management, beta and prod) run as Lima VMs on one Mac, and Rancher manages all three. The layer is spread out: the batch scheduler runs on prod with the products, the catalog and the upgrade controller run on the management cluster, and the config reloader runs on beta and prod, where the product workloads are.
The services in this layer
Apache Airflow runs the products' scheduled pipelines as Python DAGs. One shared installation on prod has its own namespace: one scheduler, one webserver, one triggerer and a Postgres metadata database of its own, with the Celery executor and a separate Redis as the Celery broker. Each product has its own Celery worker in its own namespace and image, listening on its own queue (eightball, paiddaily, sagedaily, plus platform for shared jobs). DAG files sync from the main branch of each product repository. It runs Airflow 2.10.4.
Backstage is the developer portal on the management cluster. Its software catalog lists every service in the stack, and each service's TechDocs render on its catalog page, so one page gives a service's owner, its dependencies and its documentation. Users sign in through Keycloak. It runs Backstage 1.55.0.
Reloader from Stakater runs on beta and prod. It watches ConfigMaps and Secrets and rolls the Deployments that reference them. Reloading is annotation-driven, so only workloads that opt in are rolled. Its main job here is the last step of secret rotation: External Secrets copies a rotated value from OpenBao into a Kubernetes Secret, and Reloader makes the pods pick it up. It runs v1.4.22.
The System Upgrade Controller is Rancher's node upgrade controller, v0.20.1, on the management cluster. Rancher writes its Plan objects to upgrade K3s on the clusters it manages, and each node is cordoned, drained and upgraded one at a time. No Plans are written by hand.
How the pieces fit
Three flows cross this layer. A scheduled task moves from the Airflow scheduler to a product worker. A rotated secret moves from External Secrets through Reloader into new pods. A K3s upgrade moves from Rancher through the upgrade controller to the nodes, and the app pods on those nodes stop and start again on the way. Backstage sits beside all three as the place to look up what each service is and who owns it.
Step 1 of 3. At the schedule time the scheduler creates a DAG run and sends each due task to the Celery queue named for its product. Only the worker that listens on that queue takes it.
Airflow runs on prod, Reloader on beta and prod, and the upgrade controller with Rancher on the management cluster.
The flows touch the same workloads at different times. Reloader restarts a pod because its config changed; the upgrade controller restarts it because its node changed. Both use normal Kubernetes rollout and eviction, so a Deployment with one replica has a short gap in both cases. Airflow work is separate from both: its workers run in their own namespaces, and a broken product image stops only that product's tasks.
Choices and trade-offs
- One shared Airflow. In October 2026 one installation replaced the standalone Airflow that each product used to run. One scheduler and one metadata database are less to run, patch and back up. The cost is a shared failure point: a scheduler outage pauses every product's scheduled work.
- One worker per product. Each product's dependencies stay in its own image, and its tasks run on its own queue. The Airflow version, providers and settings must still match in every image, because the scheduler parses the DAG and the worker runs it.
- Airflow kept apart from the app data. Its metadata database is its own Postgres, and its Celery broker is its own Redis. A heavy Airflow day does not load the product databases or the application cache.
- A known gap. Airflow 2.10.4 is on the 2.x line, which reached end of life on 22 April 2026 and gets no more security fixes. The move to Airflow 3 is open work: the DAGs need checking against the Airflow 3 lint rules, tasks that read the metadata database directly need rewriting, and the DAG processor and API server become separate components. The Airflow page has the detail.
- Opt-in reloads. Reloader rolls only annotated Deployments, so a Secret change cannot restart a workload that did not ask for it. The trade-off is that a workload without the annotation keeps an old value after a rotation until it restarts for another reason.
- Upgrades through Rancher. Rancher already manages the three clusters, so it drives the upgrade controller and hand-written Plans are not used. The order is beta, then prod, then the management cluster, which hosts Rancher and moves last. A nightly control-plane snapshot of each K3s datastore, kept 14 days, is the restore point if an upgrade fails.
- A catalog on the management cluster. Backstage sits with the other operator tools and signs users in through Keycloak like Rancher, Grafana and Argo CD. A Backstage install is its own codebase with monthly releases, so keeping it current is ongoing work.
Related
- Foundation layer
- Delivery layer
- Observability layer
- Security and identity layer
- Data layer
- Backup and recovery layer
- A Kubernetes homelab on one Mac with Rancher, K3s and Lima, the map of every service by layer and cluster