System Upgrade Controller

The System Upgrade Controller (SUC) is a small Kubernetes controller from Rancher that upgrades the nodes of the cluster it runs in. An operator writes a Plan object that says which nodes to change and to what version, and the controller runs the upgrade as Jobs on those nodes, one at a time or a few at a time. K3s and RKE2 use it to replace their own binary in place, and Rancher uses it to upgrade the K3s and RKE2 clusters it imports.

What it is

A general-purpose, Kubernetes-native controller for node upgrades. The project introduces one custom resource, the Plan (plans.upgrade.cattle.io, API upgrade.cattle.io/v1), which the README describes as "an outstanding intent to mutate nodes in your cluster". The controller has no built-in upgrade logic. It picks nodes, prepares them (cordon, drain), runs a container with host access on each one and records which nodes are done; the container image carries the upgrade itself.

The common image is rancher/k3s-upgrade (and its RKE2 sibling). It copies a new K3s binary over the old one on the host and sends SIGTERM to the running K3s process, so systemd or OpenRC restarts K3s on the new version. The same mechanism can run apt upgrades or kernel updates; the repository ships examples for Ubuntu packages and kernels.

• Licence: Apache 2.0. Free, no paid tier. Copyright Rancher Labs, now part of SUSE.
• Current release: v0.20.2 (2026-09-23), which updates Kubernetes libraries to v1.37 and moves the base image to SUSE BCI 16. v0.20.1 (2026-07-22) added Kubernetes v1.36 support and immutable GitHub releases.
• Family: part of the K3s / RKE2 / Rancher toolset. Rancher ships it as a Helm chart from its rancher/charts repository. The README pins the pairs: Rancher v2.15 uses chart 110.0.0 with SUC v0.20.1, and Rancher v2.14 uses chart 109.0.2 with SUC v0.19.1.

How it works

The controller watches Plan objects in its own namespace. For each Plan it resolves a target version, either from spec.version or by polling a spec.channel URL. It hashes the Plan spec into status.latestHash, then selects every node that matches nodeSelector and does not yet carry the label plan.upgrade.cattle.io/<plan-name>=<latestHash>. For up to concurrency of those nodes at a time, it creates a Job pinned to the node. When the Job succeeds, the controller writes that label on the node and moves to the next one. A change to the Plan spec changes the hash, so every node becomes eligible again.

The Job pod can have three stages: an optional prepare init container, then cordon or drain (run as kubectl in the pod), then the upgrade container. K3s uses the prepare stage to order server and agent nodes. The agent Plan's prepare step runs prepare server-plan, which loops until the server Plan reports no nodes in status.applying. That wait is in the k3s-upgrade image, not in the controller.

The controller keeps no state outside Kubernetes: no database, and no history beyond Plan status, the node labels and finished Jobs (removed after SYSTEM_UPGRADE_JOB_TTL_SECONDS_AFTER_FINISH, 900 seconds by default).

Rancher and the controller

For a K3s or RKE2 cluster imported into Rancher, version management is on by default (global setting imported-cluster-version-management, with a per-cluster override). With it on, Rancher deploys the system-upgrade-controller app and its Plans in cattle-system, and the cluster's Kubernetes version and upgrade strategy (control-plane and worker concurrency) are set in the Rancher UI. Turning version management off makes Rancher remove those components. For clusters Rancher provisioned itself, Rancher upgrades through its system agent instead, and the K3s docs say not to add hand-written Plans.

When to use it

Reach for it when:

• You run K3s or RKE2 and want upgrades declared in Git or a UI, not done by SSH on each node.
• You want servers upgraded before agents, one node at a time, with cordon or drain handled for you.
• Your clusters are imported into Rancher: Rancher already uses SUC, so the upgrade button is the supported path.
• You need upgrades held to a maintenance window (spec.window days, start and end time, time zone).

Skip it when:

• Your nodes are cloud-managed (EKS, GKE, AKS node pools). The provider replaces nodes; in-place binary swaps do not apply.
• You rebuild nodes from images (immutable infrastructure). Replacing the VM is cleaner than mutating it.
• You use kubeadm. Its upgrade sequence (kubeadm upgrade apply, then node upgrades) is a different workflow.
• You only need reboots after OS patches. kured covers that job alone.

At a glance

Core concepts

• Plan — the custom resource. Names nodes by label selector, a version or channel, an upgrade container and how to treat the node (cordon, drain).
• Channel — a URL that answers HTTP 302 to a release; the last path element is the version (+ becomes - for the image tag).
• latestHash — the hash of the Plan spec. A node is done for this Plan when it carries plan.upgrade.cattle.io/<plan>=<latestHash>.
• Concurrency — how many nodes a Plan touches at once.
• Prepare / upgrade containers — the init step and main step of each Job; the image supplies the logic.
• Exclusive — Plans marked exclusive never run Jobs alongside another exclusive Plan.
• Window — days and hours in which new Jobs may start; running Jobs may finish after it closes.

Key facts

Item	Value
API	upgrade.cattle.io/v1, kind Plan
Standalone namespace	system-upgrade (labelled pod-security.kubernetes.io/enforce: privileged)
Namespace under Rancher	cattle-system
Controller placement	Control-plane nodes only (node affinity in the manifest)
Job defaults	900 s active deadline, backoff limit 99, image pull Always
Channel poll interval	15 minutes (SYSTEM_UPGRADE_PLAN_POLLING_INTERVAL)
Job privileges	Host IPC, network and PID; CAP_SYS_BOOT; host root read-write at /host
Release assets	crd.yaml, system-upgrade-controller.yaml, binaries for amd64, arm64, arm

Upgrade paths for K3s compared

Aspect	SUC Plan	Rancher UI (imported)	Manual install script
Who writes the Plan	Operator	Rancher	No Plan
Node order	Server Plan, then agent Plan	Control-plane and worker concurrency settings	Operator's choice
Cordon / drain	Per Plan	Upgrade strategy setting	By hand
Best for	Clusters outside Rancher	Clusters imported into Rancher	One node, or recovery

How to integrate

Install standalone

Use the release assets, not master: the manifest on the default branch can lag the latest image tag.

VER=v0.20.2
kubectl apply \
  -f https://github.com/rancher/system-upgrade-controller/releases/download/$VER/crd.yaml \
  -f https://github.com/rancher/system-upgrade-controller/releases/download/$VER/system-upgrade-controller.yaml
kubectl -n system-upgrade rollout status deploy/system-upgrade-controller

Settings live in the default-controller-env ConfigMap. The controller reads it at start, so delete the controller pod after a change.

Server and agent Plans for K3s

Pin a version for controlled upgrades; swap version for channel: https://update.k3s.io/v1-release/channels/stable to follow stable automatically.

apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: server-plan
  namespace: system-upgrade
spec:
  concurrency: 1
  cordon: true
  nodeSelector:
    matchExpressions:
      - {key: node-role.kubernetes.io/control-plane, operator: In, values: ["true"]}
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/k3s-upgrade
  version: v1.36.5+k3s1
  window:
    days: [saturday, sunday]
    startTime: "02:00"
    endTime: "05:00"
    timeZone: UTC
---
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: agent-plan
  namespace: system-upgrade
spec:
  concurrency: 1
  cordon: true
  nodeSelector:
    matchExpressions:
      - {key: node-role.kubernetes.io/control-plane, operator: DoesNotExist}
  prepare:
    image: rancher/k3s-upgrade
    args: ["prepare", "server-plan"]
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/k3s-upgrade
  version: v1.36.5+k3s1

Watch a rollout

kubectl -n system-upgrade get plans -o wide
kubectl -n system-upgrade get jobs,pods
kubectl get nodes -L plan.upgrade.cattle.io/server-plan

On a Rancher-managed cluster, use -n cattle-system and change the version in the cluster's settings instead of editing Plans.

In the GL stack

The three K3s clusters behind builddaily.io, paiddaily.io and the other products run as Lima VMs on one Mac: a management cluster, a beta cluster and a prod cluster. All three run K3s v1.36.5+k3s1.

• Where it runs. SUC v0.20.1 runs on the management cluster, the version that ships with Rancher v2.15. Rancher v2.15.2 uses it to run K3s upgrades on the clusters it manages. No Plans are written by hand.
• How an upgrade moves. Rancher writes the Plan objects, and each node is cordoned and drained and upgraded one at a time.
• Order. Upgrades go to the beta cluster first, then prod, then the management cluster. Beta takes any breakage before prod does, and the management cluster, which hosts Rancher itself, moves last.
• Pinning. After all three clusters are on the new release, the same K3s version is pinned in the VM template, so a VM rebuilt from the template comes up at the version already running.
• Safety net. A nightly control-plane snapshot (a SQLite .backup of the K3s datastore plus the token, credentials and TLS files) goes to the backup store and is kept 14 days, so a failed upgrade has a recent restore point.

Gotchas

• No version-skew guard. The K3s docs state that the controller does not protect against unsupported Kubernetes version changes. Do not skip a minor version in one Plan.
• Downgrades fail and leave nodes cordoned. The k3s-upgrade image refuses a lower version or an older build date. The Job errors, and nodes with cordon: true stay unschedulable. Fix the Plan to a version at or above the running one, or delete the Plan and kubectl uncordon each node.
• Drain can hang small clusters. On one or two nodes, PodDisruptionBudgets can block eviction forever. The README points to drain.disableEviction or drain.skipWaitForDeleteTimeout; or use cordon: true with no drain.
• Job deadline. Jobs stop after 900 seconds by default. A slow image pull plus a long drain can exceed it, and the Plan then stops until it is changed. Raise jobActiveDeadlineSecs on the Plan.
• Hand-written Plans on Rancher clusters. On an imported K3s or RKE2 cluster with version management on, Rancher owns the Plans. The K3s docs say not to write Plans by hand there unless version management is turned off for that cluster.
• The agent wait is in the image. The server-before-agent order only works with the k3s-upgrade (or RKE2) image's prepare step. A custom image does not inherit it.
• ConfigMap changes need a restart. Edits to default-controller-env do nothing until the controller pod is deleted.
• A changed Secret re-runs the Plan. Secrets listed in spec.secrets are hashed into the Plan. Set ignoreUpdates: true on a Secret that should not trigger a new rollout.

Risks

• Privilege. Every upgrade Job is close to root on the host: host namespaces, CAP_SYS_BOOT, writable /. Anyone who can create a Plan in the controller's namespace can run any image as root on matching nodes. Restrict write access to Plans as tightly as cluster-admin.
• Policy engines. Admission policies that block privileged pods or host paths will block upgrades. The namespace needs an exception (the manifest sets Pod Security privileged).
• Project health. Maintained by SUSE's Rancher team, with automated dependency updates and a chart version per Rancher release line. It is a narrow project with one vendor; its life follows K3s, RKE2 and Rancher.
• Lock-in. Low. Plans are small YAML objects, and K3s can always be upgraded by rerunning the install script with INSTALL_K3S_VERSION.
• Upgrade itself. The controller makes upgrades easy to start, which makes it easy to upgrade prod by mistake with a channel Plan. Pin version on production clusters.

Alternatives

• Rancher UI version management — the right choice for any K3s or RKE2 cluster imported into Rancher; it drives SUC for you.
• K3s install script — rerun curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=<version> sh - on each node. Fine for one node or a recovery.
• kured — reboots nodes safely after OS patches, one at a time. Pick it when the job is reboots, not version changes.
• Rebuild from image — replace VMs from a template at the new version (Cluster API, Terraform, Lima templates). Pick it when nodes are disposable.

Related

• K3s and Lima — the distribution SUC upgrades and the VMs it runs on.
• Rancher — writes the Plans and sets the version for each managed cluster.
• Fleet — Rancher's GitOps engine, which also delivers Rancher's agent bundles.
• Kyverno — an admission policy engine; any such engine needs an exception for privileged upgrade Jobs.
• Velero and Rancher Backup — backups to take before an upgrade.
• Operations layer — the stack layer SUC belongs to; Foundation layer covers the clusters it upgrades.
• Official: rancher/system-upgrade-controller, Plan API reference, K3s automated upgrades, k3s-upgrade image, Registering K3s and RKE2 clusters in Rancher.