k3s Node Upgrades Without Downtime

The Install Script Never Asked Your Pods How They Felt

You re-run the k3s install script, the node restarts, and you hope the pods on it were fine with that. Nobody asked them. The script did not drain anything. It downloads a binary, rewrites a systemd unit, and restarts the k3s service. That is the whole job.

So “upgrade k3s” is a drain problem, not a k3s problem. The sequence is what matters: cordon, drain, upgrade, verify, uncordon, servers first, one node at a time. The system-upgrade-controller (SUC) is a robot that runs that same sequence for you using Plan CRDs. Learn the manual version first. Then the robot’s YAML reads like a checklist you already know.

As of October 2026, the k3s stable channel points at v1.36.5+k3s1 and the latest channel at v1.37.1+k3s1. Both were released on 2026-09-30, and v1.35 and v1.34 got patch releases the same day.

What a Bare Script Re-run Actually Does

Pod containers run under containerd, which is a separate process from the k3s service. When k3s restarts, your containers keep running. On a single-node cluster, that means the API server is gone for roughly 30 to 60 seconds (my mini PCs, your hardware will differ), but your apps keep answering requests. kubectl times out, and nothing else notices.

curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=stable sh -s - server

For a one-node home lab, that is fine. Run it, wait a minute, check kubectl get nodes, go back to your coffee.

Two gotchas. First, re-running the script without your original env vars and arguments loses them, because the script rewrites the unit file. Put your settings in /etc/rancher/k3s/config.yaml instead. The script does not manage that file, so it survives every upgrade. Second, INSTALL_K3S_VERSION=v1.36.5+k3s1 pins an exact version when you don’t want the channel deciding for you.

Multi-node clusters are where it bites. The restart itself is usually harmless, but the upgrade is a gamble you took with pods still on the node. If the new binary fails to start, or a config flag was removed in the new minor, those pods are stranded on a broken node and nothing evicted them first. Run the script on all three nodes in a loop and every API server restarts within the same minute. You skipped the drain, and a forklift driver who skips the “clear the aisle” step will eventually meet a couch.

Do not reach for k3s-killall.sh here. That script stops all containers too, which is the one that causes pod downtime. It exists for uninstalls and debugging.

Step Zero: Take a Snapshot

Before touching a server, save the datastore.

# embedded etcd (multi-server clusters)
sudo k3s etcd-snapshot save --name pre-upgrade

# single server with the default sqlite datastore
sudo systemctl stop k3s
sudo cp -a /var/lib/rancher/k3s/server/db/ /root/k3s-db-backup/
sudo cp /var/lib/rancher/k3s/server/token /root/k3s-db-backup/
sudo systemctl start k3s

The etcd command only applies to embedded etcd clusters. On the default sqlite setup, back up /var/lib/rancher/k3s/server/db/ and the token file next to it, because a restored datastore is useless without the token that encrypted it. Stopping the service first gives you a consistent copy. Yes, that is a short outage for the API. Your 2 AM self will thank you when the upgrade goes sideways.

The Manual Sequence, as a Script

The k3s docs say it plainly: upgrade server nodes first, one at a time, then agents. The loop below assumes a 3-node cluster with all three nodes as servers on embedded etcd: node-a, node-b, node-c at 192.168.1.x addresses. Run it from your workstation with working kubectl and SSH access.

#!/usr/bin/env bash
set -euo pipefail

VERSION="${1:?usage: $0 v1.36.5+k3s1}"
# order matters: servers first, one at a time
NODES=("node-a:192.168.1.11" "node-b:192.168.1.12" "node-c:192.168.1.13")

for entry in "${NODES[@]}"; do
  node="${entry%%:*}"
  ip="${entry##*:}"
  echo "=== $node ($ip) -> $VERSION ==="

  kubectl cordon "$node"
  kubectl drain "$node" \
    --ignore-daemonsets \
    --delete-emptydir-data \
    --timeout=300s

  # config.yaml on the node already holds token, tls-san, etc.
  ssh "$ip" "curl -sfL https://get.k3s.io | \
    INSTALL_K3S_VERSION='$VERSION' sh -s - server"

  # wait for the node to come back Ready on the new version
  kubectl wait --for=condition=Ready "node/$node" --timeout=300s
  kubectl get node "$node" -o wide

  kubectl uncordon "$node"
  sleep 30   # let pods reschedule before touching the next node
done

For agent-only nodes, change the last argument from server to agent. The server: and token: keys live in the agent’s /etc/rancher/k3s/config.yaml, so the script does not need them as env vars.

One catch: the API server on the node you are upgrading is down for those 30 to 60 seconds. If your kubectl kubeconfig points at that node’s IP, kubectl wait fails. Point your kubeconfig at a different server, or at a load balancer or virtual IP in front of all three.

After the loop, verify:

kubectl get nodes -o wide

The VERSION column should show the new version on every node, and no node should say SchedulingDisabled.

The Flags You Keep Googling

--ignore-daemonsets is required because DaemonSet pods are pinned to every node by design, and drain refuses to evict them. It skips them and moves on. They keep running through the restart.

--delete-emptydir-data is required if any pod on the node uses an emptyDir volume. Drain stops and complains otherwise, because that data dies with the pod. If you stash anything valuable in an emptyDir, fix that before the upgrade, not during it.

Drain respects each pod’s terminationGracePeriodSeconds and every PodDisruptionBudget. That second part is where drains go to hang forever.

Pods with a hard-coded spec.nodeName ignore cordon, because they skip the scheduler entirely. They are rare, but a controller that stamps nodeName on its pods will put a fresh one back on the cordoned node right after drain evicts the old one.

PDBs: The Deadlock You Build Yourself

A PodDisruptionBudget tells drain how many replicas it may take down at once. Good idea, easy to get wrong. The classic deadlock is a Deployment with one replica and a PDB that says minAvailable: 1. Drain wants to evict the only pod. The PDB says you can’t. Drain waits. And waits.

# This will block every drain of the node running the pod.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: myapp
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: myapp
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 1   # <- the problem
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
        - name: myapp
          image: nginx:stable

maxUnavailable: 1 on a single-replica Deployment lets the drain through, but it evicts your only pod, so the app is down until it lands somewhere else. The fix is replicas. “Zero downtime” for a workload means at least two replicas spread across different nodes, plus a PDB that protects them.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: myapp
spec:
  maxUnavailable: 1
  unhealthyPodEvictionPolicy: AlwaysAllow
  selector:
    matchLabels:
      app: myapp

The unhealthyPodEvictionPolicy: AlwaysAllow line is the one Kubernetes recommends. Without it, a crashlooping pod that never becomes Ready still counts against the budget, and it can block a drain forever. With it, a sick pod can always be evicted. Add it to every PDB you write.

To find the offenders before an upgrade, run kubectl get pdb -A and look at the ALLOWED DISRUPTIONS column. Anything showing 0 will block a drain.

Longhorn Has Its Own Opinion

If you run Longhorn, drain has a second gatekeeper. Longhorn has a Node Drain Policy setting with these values:

• block-if-contains-last-replica (the default)
• allow-if-replica-is-stopped
• always-allow
• block-for-eviction
• block-for-eviction-if-contains-last-replica

Under the default, a node holding the last healthy replica of any volume blocks the drain, because evicting would take the only good copy offline. The cheap fix is replica count 2 or 3 on volumes that matter. The loud fix is always-allow, which lets the drain proceed, and the Longhorn docs warn of data corruption if that last replica was running during the drain. block-for-eviction-if-contains-last-replica is the middle ground: Longhorn rebuilds a copy elsewhere first, then lets the drain through, at the cost of a slow drain. Those names and the default come from the Longhorn 1.11 settings reference.

Version Skew: One Minor at a Time

Kubernetes does not support hopping minor versions. Going from v1.35 to v1.37 means passing through v1.36 first, with every node upgraded at each stop. The SUC does not enforce the skew policy for you. It will happily try the skip and leave you with a cluster in an unsupported state.

The path, as of October 2026:

1. Latest v1.35 patch, on all nodes. Verify with kubectl get nodes -o wide.
2. v1.36.5+k3s1 on all nodes. Verify again.
3. v1.37.1+k3s1 on all nodes. Verify again.

With the manual script, that is three runs with a different version argument. With SUC, it is three edits to version: in your Plans, applied one at a time. Wait for each to finish.

Now Let the Robot Do It

SUC runs the same sequence inside the cluster. Install it with two manifests:

kubectl apply \
  -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/crd.yaml \
  -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/system-upgrade-controller.yaml

As of October 2026, the latest release is v0.20.2 (2026-09-23). It creates the system-upgrade namespace and a controller deployment. Then you describe the upgrade with Plans. Plans must live in the system-upgrade namespace, and they should use the system-upgrade service account, which carries the ClusterRole that lets the job cordon and drain. The upgrade image is rancher/k3s-upgrade.

You need two Plans: servers first, agents second. The agent Plan has a prepare: step that makes it wait for the server Plan to finish.

apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: server-plan
  namespace: system-upgrade
spec:
  concurrency: 1
  serviceAccountName: system-upgrade
  version: v1.36.5+k3s1
  nodeSelector:
    matchExpressions:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule
    - key: CriticalAddonsOnly
      operator: Exists
  drain:
    timeout: 300s
    ignoreDaemonSets: true
    deleteEmptydirData: true
  window:
    days: [sat, sun]
    startTime: "02:00"
    endTime: "05:00"
    timeZone: UTC
  upgrade:
    image: rancher/k3s-upgrade

apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: agent-plan
  namespace: system-upgrade
spec:
  concurrency: 1
  serviceAccountName: system-upgrade
  version: v1.36.5+k3s1
  nodeSelector:
    matchExpressions:
      - key: node-role.kubernetes.io/control-plane
        operator: DoesNotExist
  prepare:
    image: rancher/k3s-upgrade
    args: ["prepare", "server-plan"]
  drain:
    timeout: 300s
    ignoreDaemonSets: true
    deleteEmptydirData: true
  window:
    days: [sat, sun]
    startTime: "02:00"
    endTime: "05:00"
    timeZone: UTC
  upgrade:
    image: rancher/k3s-upgrade

Walk through the choices. concurrency: 1 is one node at a time, the same as the loop in the script. The drain: block maps to the kubectl drain flags you already know. DrainSpec also supports gracePeriod, force, disableEviction, skipWaitForDeleteTimeout, and podSelector. Leave force and disableEviction alone unless you enjoy bypassing your own PDBs.

The window: block uses the same format as kured. Plans only start new node upgrades inside the window, so a weekday morning won’t surprise you. The controller already caps each job at 900 seconds through its ConfigMap, and jobActiveDeadlineSecs on a Plan overrides that per Plan.

I pinned version: instead of pointing at the stable channel. A channel Plan can use channel: https://update.k3s.io/v1-release/channels/stable instead, and the controller resolves it. That is convenient, but when the channel moves from v1.36 to v1.37 your cluster follows on its own schedule. For anything you care about, a pinned version means upgrades happen when you edit a file.

Apply the Plans and watch:

kubectl apply -f server-plan.yaml -f agent-plan.yaml
kubectl -n system-upgrade get plans,jobs
kubectl get nodes -o wide

When It Fails

The rancher/k3s-upgrade image refuses downgrades. If you pin an older version than a node already runs, the job fails. It also leaves the node cordoned on failure, so the cluster stops scheduling there until you step in.

You have two ways out. Retarget the Plan to a version that makes sense, or delete the Plan and uncordon the node yourself:

kubectl -n system-upgrade get jobs,pods
kubectl -n system-upgrade logs <failed-apply-pod-name>
kubectl -n system-upgrade delete plan server-plan
kubectl uncordon node-b

If a drain hangs, a PDB or a Longhorn volume is almost always the reason. Run kubectl get pdb -A and check Longhorn’s UI for volumes with one replica on that node.

The SumGuy Take

For a single node, run the install script, wait a minute, and move on. SUC is overkill for a lone machine, like hiring a forklift to move a couch. Technically it works, but your neighbors will have questions.

For three or more nodes, learn the manual script first, then adopt SUC once you trust it. Pin the version, set a window, and write PDBs with AlwaysAllow on anything that has replicas. The robot is only as good as the cluster you hand it, and a cluster with single-replica Deployments and minAvailable: 1 PDBs will hang under SUC exactly the way it hangs under bash.

The most useful habit is boring: snapshot, upgrade one minor version, verify, wait a day, repeat. Skipping versions to save an afternoon costs a weekend.

Common Questions

Can I upgrade k3s on a single node without downtime?

No, not for the API. A single-server k3s cluster loses its API server for roughly 30 to 60 seconds while the service restarts. Running pods keep serving traffic during that gap because containerd runs separately. Workloads stay up if they don’t need the API during the restart.

Do I need system-upgrade-controller to upgrade k3s?

No. The install script with a pinned INSTALL_K3S_VERSION upgrades a node by itself, and a bash loop with kubectl drain handles multi-node clusters. System-upgrade-controller automates that same sequence with Plans, a maintenance window, and a drain block. It earns its place at three or more nodes.

Can I skip a minor version when upgrading k3s?

No. Move one minor version at a time, for example v1.35 to v1.36 to v1.37, with every node upgraded at each step. The system-upgrade-controller does not enforce the Kubernetes version skew policy, so a Plan that skips a minor version will run and can leave the cluster in an unsupported state.

How do I roll back a failed k3s upgrade?

Reinstall the old binary with INSTALL_K3S_VERSION, then restore the pre-upgrade snapshot with k3s server --cluster-reset --cluster-reset-restore-path=<snapshot>. Kubernetes does not support control-plane downgrades, and the rancher/k3s-upgrade image refuses them, so a Plan cannot roll back. For a node left cordoned after a failed job, delete the Plan and run kubectl uncordon <node>.

Related Reading

• k3s Backups: etcd Snapshots + Velero
• k3s Cluster on 3 Mini PCs From Zero
• Longhorn vs OpenEBS for k3s Storage
• Cluster API for Homelab k3s
• K3s vs K0s vs MicroK8s: Lightweight Kubernetes for Home Labs