kube-vip: HA k3s Control Plane

It’s 2 AM and Server 1 Just Rebooted

You built a three-server k3s cluster with embedded etcd. You read the HA docs. You feel safe. Then server 1 reboots for a kernel update, and kubectl starts timing out. The new worker you were adding refuses to join. Your CI job, which deploys through the API, goes red.

Nothing is wrong with the cluster. Servers 2 and 3 are fine, and etcd still has quorum. The problem is that your kubeconfig says https://192.168.1.11:6443, and that address belongs to the machine that is rebooting. Embedded etcd makes the datastore highly available. It does nothing about the one IP that every client and every joining node has been told to use.

kube-vip fixes that. It gives the Kubernetes API one floating address (a virtual IP, or VIP) that moves between your servers. You need no extra machines, because kube-vip runs as a DaemonSet on the servers you already have. The classic alternative is a pair of keepalived plus HAProxy boxes, which is what the k3s docs show first. This post builds the kube-vip version end to end, measures what happens when you pull the plug, and compares the two approaches.

Full example: Clone the working files at github.com/KingPin/sumguy-examples/devops/kube-vip-k3s-ha-control-plane

If you have not built the cluster yet, start with k3s on three mini PCs from zero. This post assumes three servers already exist, or that you are about to create them.

What the VIP Actually Protects

k3s agents already run a client-side load balancer. After an agent registers, it learns the addresses of all the servers and keeps connections to them. If one server dies, an existing agent keeps working.

So the VIP does not keep your workers alive. It protects four other things:

1. Your kubeconfig on your laptop.
2. CI jobs, Argo CD, Flux, and anything else that calls the API from outside.
3. New or rebuilt nodes that need to register while server 1 is down.
4. The server: line in every agent config, which has to name something.

That is still a real list. Many guides skip the part about agents, and it matters: if your worker fleet runs fine today without a VIP, the VIP mostly serves you and your tooling. The pods were already covered.

The Requirements

kube-vip in ARP mode is simple, and it has three requirements:

• Three servers (any odd number of 3 or more works for etcd quorum, which is (n/2)+1).
• All servers on the same layer 2 network or VLAN. ARP does not cross routers.
• A free IP address for the VIP, outside your router’s DHCP range.

In this post the VIP is 192.168.1.50 and the servers are 192.168.1.11, .12, and .13. Pick a VIP that nothing else uses. If your router hands out .50 to a phone, you will have two machines answering for one address, and the failures look random.

Why a DaemonSet and Not a Static Pod

The kube-vip docs offer both. On k3s you use a DaemonSet. k3s can bootstrap a single server before any load balancer exists, and it auto-deploys any manifest placed in /var/lib/rancher/k3s/server/manifests/. That directory is where the RBAC file and the DaemonSet go. No kubectl apply needed.

Step 1: Pick the Interface and Generate the Manifest

Find your interface name on a server:

ip -br addr

lo               UNKNOWN        127.0.0.1/8 ::1/128
eth0             UP             192.168.1.11/24

Here the interface is eth0. Your name may be enp3s0, eno1, or something else. Use the real one.

The kube-vip image generates its own manifest. The kit wraps that in a small script that runs the image with Docker, so you can do it on any machine, even before k3s exists:

#!/usr/bin/env bash
set -euo pipefail
: "${VIP:?set VIP, e.g. VIP=192.168.1.50}"
: "${INTERFACE:?set INTERFACE, e.g. INTERFACE=eth0 (check with: ip -br addr)}"
KVVERSION="${KVVERSION:-v1.3.1}"

docker run --network host --rm "ghcr.io/kube-vip/kube-vip:${KVVERSION}" \
  manifest daemonset \
  --interface "$INTERFACE" \
  --address "$VIP" \
  --inCluster \
  --taint \
  --controlplane \
  --arp \
  --leaderElection

Run it:

VIP=192.168.1.50 INTERFACE=eth0 ./generate-manifest.sh > kube-vip.yaml

What the flags do:

• --inCluster makes kube-vip use its ServiceAccount to talk to the API.
• --taint adds the control-plane node affinity and tolerations, so the pods run only on servers.
• --controlplane turns on the API VIP feature.
• --arp picks ARP mode (more on that below).
• --leaderElection makes the servers elect one holder through a Kubernetes lease.

Version note, checked 2026-11-10: use kube-vip v1.3.1 (released 2026-10-19). The maintainers marked v1.3.0 “do not install” because its binary reported v1.2.4 as its version. Skip v1.3.0. That release also restricted shared VIPs to Services on the same lease. This only matters if you use kube-vip for Services, which this post does not.

What the Generated Manifest Contains

The full file is about 90 lines. These env entries are the ones worth reading:

        env:
        - name: vip_arp
          value: "true"
        - name: port
          value: "6443"
        - name: vip_interface
          value: eth0
        - name: vip_subnet
          value: "32"
        - name: cp_enable
          value: "true"
        - name: vip_leaderelection
          value: "true"
        - name: vip_leasename
          value: plndr-cp-lock
        - name: vip_leaseduration
          value: "15"
        - name: vip_renewdeadline
          value: "10"
        - name: vip_retryperiod
          value: "2"
        - name: address
          value: 192.168.1.50

The container drops all capabilities and adds back NET_ADMIN and NET_RAW. It needs those to put an address on a network interface and to send ARP packets. It also sets hostNetwork: true, because the VIP has to land on the host’s real NIC.

The node affinity matches either node-role.kubernetes.io/master or node-role.kubernetes.io/control-plane. That matters on current k3s. In our lab, k3s v1.37.2 servers showed the roles control-plane,etcd and no master label. The affinity accepts either one, so the pods still schedule.

Step 2: Configure the First Server

A common failure with this setup is a certificate error. The k3s API server certificate lists the node’s own IPs automatically. It does not list your VIP. The kube-vip docs say the --tls-san flag must carry the same IP when you install the first server. In a config file that is the tls-san key:

# The VIP must be in tls-san, or kubectl through the VIP fails certificate checks.
cluster-init: true
token: "change-me-to-a-long-random-string"
tls-san:
  - 192.168.1.50
  - k3s.home.arpa

Put the manifests in place before you start k3s, so k3s deploys them on first boot:

sudo mkdir -p /var/lib/rancher/k3s/server/manifests
sudo curl -o /var/lib/rancher/k3s/server/manifests/kube-vip-rbac.yaml \
  https://kube-vip.io/manifests/rbac.yaml
sudo cp kube-vip.yaml /var/lib/rancher/k3s/server/manifests/kube-vip.yaml
curl -sfL https://get.k3s.io | sh -

The manifests go only on servers. Agents have no manifests directory worth using, and the DaemonSet affinity would keep the pods off them anyway.

Step 3: Join Servers 2 and 3

Here the 2 AM lesson shows up early. If you start servers 2 and 3 before server 1 is serving, they exit with this error:

failed to validate token: failed to get CA certs

Start server 1 first. Wait until kubectl get nodes answers on it. Then start the others. Our lab script does exactly this, with a polling loop between the steps.

The join config points at server 1’s real address:

# Join through a real server IP. The VIP works too once kube-vip is up,
# but server 1's address has no dependency on kube-vip at all.
server: https://192.168.1.11:6443
token: "change-me-to-a-long-random-string"
tls-san:
  - 192.168.1.50
  - k3s.home.arpa

Joining through the VIP also works once kube-vip is running, but a join through a real server has one less dependency during bootstrap. The manifests only need to exist on server 1; k3s applies them to the whole cluster. If you also copy them to servers 2 and 3 (so a rebuilt server 1 is not the only copy), keep the files identical on all three. Both joining servers need the same tls-san list. Each server builds the SAN list for its 6443 certificate from its own config, and the k3s HA docs pass --tls-san to every server.

Step 4: Point Agents and Your Kubeconfig at the VIP

Agents register through the VIP, so a dead server 1 never blocks a new node:

server: https://192.168.1.50:6443
token: "change-me-to-a-long-random-string"

Install agents with curl -sfL https://get.k3s.io | sh -s - agent. The bare sh - from step 2 installs a server when K3S_URL is unset, and with this config the node would join the control plane as a fourth etcd member.

Your kubeconfig still says 127.0.0.1 or server 1’s IP. Fix it:

sed -i -E 's#https://(127\.0\.0\.1|192\.168\.1\.11):6443#https://192.168.1.50:6443#' ~/.kube/config
kubectl get nodes

Forgetting this step is another common mistake. The VIP can be perfect and your kubectl still talks to server 1.

Checking Who Holds the VIP

The lease tells you:

kubectl -n kube-system get lease plndr-cp-lock -o jsonpath='{.spec.holderIdentity}{"\n"}'

kv-server1

In ARP mode, only the holder puts the VIP on its interface and answers ARP requests for it. It broadcasts a gratuitous ARP every 3000 ms by default (the vip_arpRate setting). The consequence is that one server takes all API traffic that goes through the VIP. This is failover, not load balancing. kube-vip does have an optional IPVS control-plane load balancer (lb_enable, --enableLoadBalancer), but this post does not use it.

There is also a BGP mode, where every node advertises the VIP to your router. It needs a router that speaks BGP. Most consumer routers do not, so most home labs use ARP.

Pulling the Plug: What Our Lab Measured

We tested failover in a lab. Read the limits first: this was a 3-server k3s v1.37.2 cluster in Docker containers on one desktop’s bridge network. It was not real hardware, so real switches and clients will behave differently.

The test script (lab/failover-test.sh in the kit) brings up the cluster, finds the lease holder, and runs docker kill on it. That is the container version of pulling the power cord. Then it polls https://VIP:6443/livez in a loop (1 second curl timeout, 0.2 second sleep) until something answers.

A note on the probe. On k3s, curl -sk https://VIP:6443/livez returns HTTP 401, because anonymous requests are rejected. That still proves the API server answered, so the script counts any HTTP code as up.

Eight runs completed the failover step. The VIP stopped answering for these times:

The median is 18.1 seconds, and six of the eight runs came in under 22 seconds. Run 6 took 48.3 seconds. We did not trace why, so treat it as an observation: plan for the occasional long gap. Server 1 held the lease at the start of every run, and server 2 or server 3 took it over each time. One more run (run 2) failed during cluster start-up, before the failover step, also untraced. Every run is in the raw results file.

The 15 second lease duration sets the floor. The survivors wait for the dead leader’s lease to expire before one of them takes over. Read the result as “usually 15 to 25 seconds on a LAN, with occasional longer gaps.” On real hardware, add the time your switch and your clients take to notice that the VIP’s MAC address changed.

The same run also confirmed that the serving certificate listed the VIP, which is the tls-san setting doing its job.

The Alternative: keepalived Plus HAProxy

The k3s docs on cluster load balancers lead with a different design: two extra machines, lb-1 and lb-2, each running HAProxy and keepalived. HAProxy listens on 6443 and spreads connections across all three API servers. keepalived floats a VIP between the two load balancer boxes using VRRP.

The HAProxy config from the k3s docs:

frontend k3s-frontend
    bind *:6443
    mode tcp
    option tcplog
    default_backend k3s-backend

backend k3s-backend
    mode tcp
    option tcp-check
    balance roundrobin
    default-server inter 10s downinter 5s
    server server-1 10.10.10.50:6443 check
    server server-2 10.10.10.51:6443 check
    server server-3 10.10.10.52:6443 check

And the keepalived config:

global_defs {
  enable_script_security
  script_user root
}

vrrp_script chk_haproxy {
    script 'killall -0 haproxy' # faster than pidof
    interval 2
}

vrrp_instance haproxy-vip {
    interface eth1
    state <STATE> # MASTER on lb-1, BACKUP on lb-2
    priority <PRIORITY> # 200 on lb-1, 100 on lb-2
    virtual_router_id 51
    virtual_ipaddress {
        10.10.10.100/24
    }
    track_script {
        chk_haproxy
    }
}

On lb-1 you set state MASTER and priority 200. On lb-2 you set BACKUP and 100. If HAProxy dies on the master, the check script fails and the VIP moves.

That design has real strengths:

• It spreads API requests round-robin across all three servers, so you get real load balancing.
• It health-checks each backend, every 10 seconds by default in this config.
• It lives outside the cluster. If the cluster is unhealthy, the load balancer does not care.

It also has costs. You need two more machines or VMs. You can run HAProxy on the servers themselves, but then bind *:6443 collides with the API server on the same port, and you must move one of them. That is fiddlier than it sounds.

The k3s docs also show nginx as a single load balancer and warn that it reintroduces a single point of failure. They list kube-vip as an option, and they say that for LoadBalancer Services, MetalLB or kube-vip is usually a better choice than an external load balancer.

The Gotchas, With Fixes

1. x509 error through the VIP. The certificate has no VIP in it. Add tls-san to /etc/rancher/k3s/config.yaml on every server. Then run systemctl stop k3s, k3s certificate rotate, and systemctl start k3s, which is the fix the k3s docs give.
2. Wrong interface name. The DaemonSet pods start but the VIP never appears. Run ip -br addr and fix vip_interface. If your servers have different NIC names, delete the vip_interface entry from the manifest. kube-vip v1.3.1 then uses the interface that holds the default route on each server. One manifest per server does not work, because every server deploys the same DaemonSet.
3. VIP inside the DHCP range. Two machines answer for one address. Reserve the VIP outside the pool.
4. Manifest on an agent. It goes only in the server manifests directory. The affinity keeps the pods on servers anyway, but a stray file on an agent is confusing noise.
5. Kubeconfig still pointing at server 1. Use the sed line above.
6. kube-vip --services next to ServiceLB or MetalLB. Two controllers will fight over the same Services. This kit is control-plane only, with no --services flag. Keep Service load balancing with ServiceLB or MetalLB. See MetalLB on bare metal for that side. If you do want kube-vip for Services, k3s needs --disable servicelb.

What kube-vip Cannot Save You From

If you lose two of three servers, etcd loses quorum. The datastore cannot commit writes, so kube-vip cannot renew or take its lease either, and the VIP stops answering. No VIP solution fixes that. Plan for it with backups. k3s backups with etcd snapshots and Velero covers that, and etcd vs Consul vs ZooKeeper explains why the quorum math works the way it does.

kube-vip also depends on the API server and the lease to elect a leader. keepalived plus HAProxy do not. That is the trade: you remove two machines and accept a dependency on the cluster you are protecting.

Rolling upgrades are the everyday version of this: a server reboot moves the VIP, which is the point. k3s node upgrades without downtime shows the order.

Which One to Pick

For most home labs, pick kube-vip. You get a floating API address with zero extra hardware, one DaemonSet, and two files in a manifests directory. A 15 to 25 second blip (sometimes longer) during a server failure is fine when the person it affects is you.

Pick keepalived plus HAProxy if you already run those boxes for other services, or if you want real round-robin balancing across all three API servers from outside the cluster. Also pick it if you need the load balancer to work while the cluster is broken. Spinning up two VMs just to float one IP is a forklift for a couch, though. Technically it works, but your power bill will have questions.

Common Questions

Do I need kube-vip if my k3s agents already load balance?

Yes, if you want a stable address for kubectl, CI, and new nodes. The agent load balancer keeps existing workers connected to all servers. It does nothing for your kubeconfig or for a node registering while server 1 is down. kube-vip covers those cases.

Why does kubectl through the kube-vip VIP fail with an x509 error?

The API server certificate does not list the VIP. k3s adds node IPs automatically but not floating addresses. Add the VIP under tls-san in /etc/rancher/k3s/config.yaml on every server. Then stop k3s, run k3s certificate rotate, and start k3s, which is the fix the k3s docs give.

How long does kube-vip failover take on k3s?

Expect about 15 to 25 seconds, with occasional longer gaps. In our Docker lab, eight kube-vip failover runs had a median of 18.1 seconds, and six finished under 22 seconds. One run took 48.3 seconds. The 15 second lease duration sets the floor. Real hardware adds switch and client ARP cache delay.

Does kube-vip load balance API traffic across all three servers?

No. In ARP mode only the lease holder answers for the VIP, so one server takes all VIP traffic. That makes kube-vip a failover tool. For round-robin across all servers, use HAProxy, or enable kube-vip’s optional IPVS control-plane load balancer.

Can I use kube-vip and MetalLB together on k3s?

Yes, if kube-vip handles only the control-plane VIP and MetalLB handles Services. Generate the kube-vip manifest without the --services flag, and disable ServiceLB with --disable servicelb before installing MetalLB. Two controllers claiming the same LoadBalancer Services will fight over the addresses, and the failures look random.

Related Reading

• k3s Cluster on 3 Mini PCs From Zero
• k3s Node Upgrades Without Downtime
• MetalLB for Bare-Metal LoadBalancer
• k3s Backups: etcd Snapshots + Velero
• HAProxy: Load Balancing Done Right