k3s Backups: etcd Snapshots + Velero

Your Cluster Backup Strategy Is Probably a Handshake Agreement

You’ve got a k3s cluster running on a couple of VMs in your home lab. It’s working great. Everything’s fine. You’re definitely not thinking about the day the primary node’s storage controller catches fire and takes the entire /var/lib/rancher/k3s directory with it.

Backing up Kubernetes isn’t optional if you’re serious about self-hosting. etcd is your cluster’s brain. Lose it, and you lose every deployment, secret, and ConfigMap you’ve ever created. The good news? k3s makes snapshots stupidly easy, and Velero turns disaster recovery from “pray the backup works” into something you can actually test.

This is a two-layer strategy: etcd snapshots (the “cluster state” layer) and Velero (the “application plus storage” layer). Two things will bite you along the way, and both get their own section below: k3s snapshots are useless without the server token, and Velero on a stock k3s cluster backs up zero bytes of your persistent volume data.

Layer 1: etcd Snapshots (The Cluster Brain Backup)

Heads up: a single-server k3s install defaults to SQLite, not etcd. You only get embedded etcd when you start the cluster with --cluster-init (or run multi-server HA). This whole layer assumes you’re on embedded etcd. If you’re on the default SQLite, the k3s etcd-snapshot commands won’t apply, and you’d back up /var/lib/rancher/k3s/server/db/ instead. Embedded etcd runs inside the k3s process itself (no separate container). Snapshots capture every last ConfigMap, deployment, secret, and RBAC rule: the entire cluster’s state at a moment in time.

Where k3s Stores Snapshots

By default, k3s writes snapshots to /var/lib/rancher/k3s/server/db/snapshots/. They’re point-in-time files, usually a few MB each. The directory tracks --data-dir, so if you moved that, the snapshots moved with it.

One more file matters as much as the snapshot: /var/lib/rancher/k3s/server/token. k3s encrypts the confidential data inside a snapshot with an AES-256 key derived from that token. Restore a snapshot with a different token value and the snapshot is unusable. Back the token up alongside your snapshots, and keep it somewhere separate from the snapshot bucket. Anyone holding both files can extract your cluster CA private keys, not just your workloads.

Set Up Automatic Snapshots

Scheduled snapshots are on by default on embedded etcd: cron 0 */12 * * * (00:00 and 12:00 system time), keeping the last 5. You can tune both:

# Install k3s with embedded etcd + snapshot tuning
# (--cluster-init switches the datastore from SQLite to embedded etcd)
curl -sfL https://get.k3s.io | sh -s - \
  --cluster-init \
  --etcd-snapshot-schedule-cron "0 */6 * * *" \
  --etcd-snapshot-retention 10

That’s every 6 hours, keep 10 snapshots. For a home lab, 12 hours with 5 snapshots is fine unless you’re changing cluster config hourly.

Manual snapshots taken with k3s etcd-snapshot save are a different animal: they have no retention at all. Nothing prunes them. Clean up with k3s etcd-snapshot delete <name> or k3s etcd-snapshot prune, or watch your snapshot directory grow forever after your third emergency “let me just grab one first”.

To see what you actually have, prefer the cluster-wide view:

# Only shows snapshots this node can see
k3s etcd-snapshot ls

# Shows every snapshot on every cluster member, local and S3
kubectl get etcdsnapshotfile

Get the Snapshots Off the Node

Snapshots land on the node that took them. If that node’s storage dies, the snapshots die with it. You need them off-node, and k3s does this itself: point it at an S3-compatible bucket and every snapshot, scheduled or on-demand, gets uploaded as it is taken.

curl -sfL https://get.k3s.io | sh -s - \
  --cluster-init \
  --etcd-snapshot-schedule-cron "0 */6 * * *" \
  --etcd-snapshot-retention 10 \
  --etcd-s3 \
  --etcd-s3-endpoint s3.us-east-1.wasabisys.com \
  --etcd-s3-bucket k3s-etcd \
  --etcd-s3-folder prod-cluster \
  --etcd-s3-region us-east-1 \
  --etcd-s3-retention 30

Three details that cost people an afternoon:

• --etcd-s3-endpoint is a bare host, not a URL. The default is s3.amazonaws.com, so write minio.internal:9000, not https://minio.internal:9000. Add --etcd-s3-insecure if that MinIO box is plain HTTP on the LAN.
• Credentials come from --etcd-s3-access-key and --etcd-s3-secret-key, but both also read AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from the environment, which keeps them out of your systemd unit.
• --etcd-s3-retention defaults to whatever --etcd-snapshot-retention is set to. Set it explicitly if you want a shallow local history and a deep remote one, as above: 10 on disk, 30 in the bucket.

If you’d rather not put credentials in a config file at all, k3s reads them from a Secret. Start the server with only --etcd-s3 --etcd-s3-config-secret=k3s-etcd-snapshot-s3-config (any other --etcd-s3-* flag makes k3s ignore the Secret entirely), then create it in kube-system:

apiVersion: v1
kind: Secret
metadata:
  name: k3s-etcd-snapshot-s3-config
  namespace: kube-system
type: etcd.k3s.cattle.io/s3-config-secret
stringData:
  etcd-s3-endpoint: "minio.internal:9000"
  etcd-s3-insecure: "true"
  etcd-s3-access-key: "AKIAEXAMPLE"
  etcd-s3-secret-key: "s3cr3t"
  etcd-s3-bucket: "k3s-etcd"
  etcd-s3-folder: "prod-cluster"
  etcd-s3-region: "us-east-1"

The catch: the Secret only works for saving, listing, and pruning. A restore happens before the apiserver is up, so there is no Secret to read. You pass the S3 flags on the command line when restoring.

Don’t have S3? Then a sync script is fine, but drop the --delete you’ll be tempted to add. With --delete, your remote copy is an exact mirror of the node, which means a node that prunes down to 5 snapshots prunes your off-node history to 5 as well. The whole point of off-node was keeping history the node no longer has.

0 * * * * rsync -a --info=stats1 \
  /var/lib/rancher/k3s/server/db/snapshots/ \
  [email protected]:/volume1/k3s-etcd/$(hostname)/ \
  >> /var/log/etcd-backup.log 2>&1

Test the Restore (You Must Do This)

Reading a disaster recovery plan isn’t the same as surviving one. Do this on a test cluster or VM, not production. The restore is a stop, reset, start sequence, and the middle step is a one-shot foreground command:

# 1. Stop k3s
sudo systemctl stop k3s

# 2. Reset the cluster to a single member and load the snapshot.
#    This runs in the foreground and exits when it's done.
sudo k3s server \
  --cluster-reset \
  --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-node1-1756900000

# 3. Wait for: "Managed etcd cluster membership has been reset,
#    restart without --cluster-reset flag now."
sudo systemctl start k3s

# 4. Verify
kubectl get all --all-namespaces
kubectl get secrets --all-namespaces

The old database is moved aside to /var/lib/rancher/k3s/server/db/etcd-old-$TIMESTAMP/, so a botched restore is recoverable. k3s also drops an empty /var/lib/rancher/k3s/server/db/reset-flag file that blocks a second reset until k3s starts normally again, which is the guard rail that stops a panicking operator from resetting three times in a row.

Restoring from S3 changes one thing: pass the S3 flags on the command line and give only the snapshot filename as the restore path, not a full path. Going the other way, if S3 config lives in your k3s config file and you want the local copy, add --etcd-s3=false and pass the full local path.

On a three-server cluster, run the reset on one server only. Then, on the other two, rm -rf /var/lib/rancher/k3s/server/db/ and start k3s again to rejoin the restored cluster. Skip that deletion and they’ll try to rejoin with a database that disagrees with the one you just restored.

Restoring to a different machine works too, and this is where the token comes back: pass --token=<the original server token> along with the restore path. Old node objects ride along inside the snapshot, so delete the stale ones once the cluster is up.

If your clusters are pets (one-off setups), run a restore drill every quarter. If you’re running fleet management, automate it as a weekly test cluster spin.

Layer 2: Velero (The Application + Storage Backup)

etcd snapshots get you the cluster config, but they don’t back up persistent volumes. If you’re running databases, Nextcloud, or anything stateful, a snapshot without Velero is like backing up the OS but leaving your home directory on the failed drive.

Velero backs up:

• Kubernetes objects (overlaps with etcd, but explicit and portable)
• Persistent volume data, if your storage cooperates (see the next section)
• Custom resources (CRDs, operators, all of it)

Install the CLI

wget https://github.com/vmware-tanzu/velero/releases/download/v1.18.1/velero-v1.18.1-linux-amd64.tar.gz
tar -xzf velero-v1.18.1-linux-amd64.tar.gz
sudo mv velero-v1.18.1-linux-amd64/velero /usr/local/bin/

velero version --client-only

Match the AWS plugin to the server version or the install silently misbehaves. Plugin v1.14.x pairs with Velero v1.18.x, v1.13.x with v1.17.x, v1.12.x with v1.16.x. The compatibility table lives in the velero-plugin-for-aws README and it changes every release.

The local-path Trap That Eats Your PV Data

Read this before you trust a green backup. k3s ships local-path-provisioner as its default StorageClass, and when neither the StorageClass nor the PVC carries a volume type annotation, local-path creates a hostPath volume. Velero’s File System Backup, the thing that actually copies bytes out of a volume, does not support hostPath. Local persistent volumes are supported; hostPath is not.

The result is a backup that completes, reports success, and contains zero bytes of your Postgres data. Nothing warns you. You find out during the restore drill you were going to skip.

Two ways out. Tell local-path to make local volumes instead:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-path
  annotations:
    defaultVolumeType: "local"
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: rancher.io/local-path
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

The StorageClass annotation applies to every volume created from it and a volumeType annotation on an individual PVC overrides it. Either way, existing volumes keep the type they were created with, so this only fixes new PVCs. Anything already running has to be migrated.

Or skip local-path for stateful workloads and use a CSI driver that does real snapshots. That’s the Longhorn or OpenEBS conversation, and for anything you actually care about losing it’s the better answer.

Two more File System Backup limits worth knowing up front: Velero uses one static encryption key for every backup repository it creates, so anyone with read access to the bucket can decrypt the contents. And FSB only reads volumes that a running pod has mounted. An orphan PVC with no pod attached gets skipped, and the usual workaround is a staging pod that mounts it and sleeps.

Configure a Backend and Install

MinIO on the NAS, or Backblaze B2, both work. MinIO first:

export MINIO_ENDPOINT="http://minio.internal:9000"
export MINIO_BUCKET="velero-backups"

cat > /tmp/credentials-minio <<EOF
[default]
aws_access_key_id = minioadmin
aws_secret_access_key = changeme-really
EOF

Then install into the cluster:

velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.14.0 \
  --bucket "$MINIO_BUCKET" \
  --secret-file /tmp/credentials-minio \
  --use-volume-snapshots=false \
  --use-node-agent \
  --default-volumes-to-fs-backup \
  --backup-location-config region=us-east-1,s3ForcePathStyle="true",s3Url="$MINIO_ENDPOINT" \
  --wait

Every flag there is load-bearing:

• --use-volume-snapshots=false because MinIO has no volume snapshot API. Leave it true and Velero creates a VolumeSnapshotLocation that never works.
• --use-node-agent deploys the node-agent DaemonSet. Without it there is no File System Backup, and PV data is not copied at all.
• --default-volumes-to-fs-backup opts every volume in by default instead of making you annotate each pod.
• s3ForcePathStyle="true" is not optional for MinIO. Miss it and the AWS SDK uses virtual-host addressing: uploads succeed and downloads fail, so you discover the problem during the restore.
• Everything MinIO-specific goes in --backup-location-config. There is no --env flag on velero install, and snapshotLocation= is not a valid config key.

Velero creates its own namespace (velero) and starts reconciling backups.

Schedule Automated Backups

# Daily backup, keep 30 days
velero schedule create k3s-daily \
  --schedule="0 2 * * *" \
  --include-namespaces "*" \
  --exclude-namespaces velero,kube-system,kube-node-lease,kube-public \
  --ttl 720h

velero schedule get

That backs up everything except system namespaces, every day at 2 AM, retained for 30 days. Each run creates a backup named k3s-daily-<timestamp>.

Verify Backup Health

velero backup get

# Was the PV data actually copied, or just the PVC object?
velero backup describe k3s-daily-20260903020017 --details
kubectl -n velero get podvolumebackups -l velero.io/backup-name=k3s-daily-20260903020017

kubectl logs -n velero deployment/velero -f

That podvolumebackups line is the one that catches the hostPath problem. No PodVolumeBackup objects means no volume data was copied, whatever the backup phase says.

Restore from Velero

This is the moment of truth. Cluster’s hosed, etcd is gone, and you need data back.

# Name the restore yourself so you can follow it
velero restore create nextcloud-dr-01 --from-backup k3s-daily-20260903020017

velero restore describe nextcloud-dr-01
velero restore logs nextcloud-dr-01

kubectl get pods --all-namespaces
kubectl get pvc --all-namespaces

Note the two different names. --from-backup takes the backup name, while describe and logs take the restore name. Let Velero auto-generate the restore name and it becomes <backup>-<timestamp>, which is where people paste the backup name into velero restore logs and get “not found”.

Restoring only one app is the same command with a filter:

velero restore create nextcloud-only-01 \
  --from-backup k3s-daily-20260903020017 \
  --include-namespaces nextcloud

Object restore is quick. Volume restore is bounded by how fast the node agent can pull data out of object storage, so a 200GB Nextcloud volume over a home uplink is an evening, not a coffee break.

Restore Drill Script

Test this quarterly. Seriously. A backup you’ve never restored from isn’t a backup; it’s hope.

The script below refuses to run without an explicit kube context, because the original version of this pattern (a variable named CLUSTER_NAME that gets echoed and then never used) restores into whatever cluster your kubeconfig currently points at. That is production, at 11pm, right after you finished a kubectl session.

#!/bin/bash
set -euo pipefail

SCHEDULE="${1:?usage: test-velero-restore.sh <schedule-name> <kube-context>}"
CONTEXT="${2:?refusing to run: pass the test cluster kube context explicitly}"
RESTORE_TIMEOUT=900

case "$CONTEXT" in
  *prod*|*production*) echo "Refusing to drill against '$CONTEXT'"; exit 1 ;;
esac

VELERO="velero --kubecontext $CONTEXT"
KUBECTL="kubectl --context $CONTEXT"

echo "Drilling schedule '$SCHEDULE' against context '$CONTEXT'"

# Backups from a schedule are named <schedule>-<UTC timestamp>,
# so a reverse lexical sort is a reverse chronological sort.
LATEST_BACKUP=$($VELERO backup get --selector "velero.io/schedule-name=$SCHEDULE" \
  | awk 'NR>1 {print $1}' | sort -r | head -1)

if [ -z "$LATEST_BACKUP" ]; then
    echo "ERROR: no backup found for schedule '$SCHEDULE'"
    exit 1
fi

echo "Restoring from: $LATEST_BACKUP"
RESTORE_NAME="drill-$(date +%s)"
$VELERO restore create "$RESTORE_NAME" --from-backup "$LATEST_BACKUP"

START=$(date +%s)
while true; do
    PHASE=$($VELERO restore describe "$RESTORE_NAME" | awk '/^Phase:/ {print $NF}')

    if [ "$PHASE" = "Completed" ]; then
        echo "Restore succeeded"
        break
    elif [ "$PHASE" = "Failed" ] || [ "$PHASE" = "PartiallyFailed" ]; then
        echo "Restore failed"
        $VELERO restore logs "$RESTORE_NAME"
        exit 1
    fi

    ELAPSED=$(($(date +%s) - START))
    if [ "$ELAPSED" -gt "$RESTORE_TIMEOUT" ]; then
        echo "Restore timeout after ${RESTORE_TIMEOUT}s"
        exit 1
    fi

    echo "  Status: $PHASE (${ELAPSED}s)"
    sleep 10
done

# Smoke test: did volume data come back, or just the objects?
PODS=$($KUBECTL get pods --all-namespaces --no-headers | wc -l)
PVCS=$($KUBECTL get pvc --all-namespaces --no-headers | wc -l)
RESTORED_VOLS=$($KUBECTL -n velero get podvolumerestores \
  -l velero.io/restore-name="$RESTORE_NAME" --no-headers 2>/dev/null | wc -l)

echo "  Pods: $PODS  PVCs: $PVCS  Volume restores: $RESTORED_VOLS"

if [ "$PODS" -gt 0 ] && [ "$PVCS" -gt 0 ] && [ "$RESTORED_VOLS" -gt 0 ]; then
    echo "Restore drill passed"
else
    echo "Restore drill FAILED: objects came back but volume data did not"
    exit 1
fi

Run it against a scratch cluster:

bash test-velero-restore.sh k3s-daily drill-cluster

The RESTORED_VOLS check is the part that matters. A drill that only counts pods and PVCs passes happily on a cluster whose volumes are all empty.

What Actually Gets Backed Up?

etcd snapshots plus Velero, configured as above:

• Cluster config (deployments, services, ConfigMaps, secrets, RBAC)
• Custom resources and operators
• StorageClass definitions
• Ingress rules, network policies
• Persistent volume data, only if the volumes are not hostPath and you passed --use-node-agent

Not backed up:

• The k3s binaries themselves (they’re ephemeral; redeploy)
• Container images (you’re using a registry, right?)
• The server token, unless you copied it yourself. k3s does not put it in the snapshot bucket for you
• Volumes with no running pod attached (FSB reads through the pod mount)
• CNI plugin data (reapply the same plugin on restore)

A Decision: When to Use What

etcd snapshots alone if:

• You need the fastest possible recovery time objective
• Your cluster is stateless: everything reconstructible from Git and container images
• You only care about cluster state, not application data

Add Velero if:

• You have persistent data (databases, files, stateful services)
• You want granular restore options, namespace by namespace
• You want to migrate a cluster (back up from A, restore to B)

Both, plus a real CSI storage class, if:

• Your cluster is your home lab’s spine and downtime costs you
• You have a multi-node cluster where pods move between hosts

That last combination is the one most home labs need and the one most home labs skip, because local-path works fine right up until the moment you need it to have been backed up.

Common Questions

Does k3s use etcd by default?

No. A single-server k3s install uses SQLite at /var/lib/rancher/k3s/server/db/. Embedded etcd appears only when you start the first server with --cluster-init or join a second server. Run k3s etcd-snapshot ls: if it errors out, you’re on SQLite and none of the snapshot flags apply.

Do I need to back up the k3s server token?

Yes, and separately from the snapshots. k3s encrypts confidential data inside every snapshot with an AES-256 key derived from /var/lib/rancher/k3s/server/token. A snapshot restored with a different token is unusable. Store the token apart from the snapshot bucket, because whoever holds both can extract your cluster CA private keys.

Does Velero back up local-path volumes on k3s?

Not by default. k3s’s local-path-provisioner creates hostPath volumes when no volume type annotation is set, and Velero’s File System Backup skips hostPath entirely. Annotate the StorageClass with defaultVolumeType: "local" for new volumes, or move stateful apps to Longhorn or another CSI driver.

Can I restore an etcd snapshot to a different machine?

Yes. Copy the snapshot file and the original server token to the new host, then run k3s server --cluster-reset --cluster-reset-restore-path=<file> --token=<original-token>. Node objects are stored inside the snapshot, so delete the stale node entries with kubectl delete node once the restored cluster is running.

How often should I run a restore drill?

Quarterly for a home lab, monthly if the cluster runs anything you’d miss for a full day. The drill is what catches an expired MinIO credential, a plugin version mismatch, or a volume that was never actually copied. Those three failures are all invisible until the day you try to restore.

The Honest Take

Setting up k3s backups feels like insurance: tedious until the day you need it. The gap between “I have backups” and “I can recover” is one afternoon of drilling, and the drill is where you learn that your PV data was never in the bucket.

Do the boring parts. Copy the server token. Check for PodVolumeBackup objects. Run the drill against a scratch cluster. Your 2 AM self will appreciate it.

Go back up that cluster.

Related Reading

• Velero: K8s Backup and DR
• Longhorn vs OpenEBS for k3s Storage
• You Should Be Testing Your Restores
• k3s Cluster on 3 Mini PCs From Zero
• ArgoCD for Home Lab GitOps