A highly available k3s control plane with keepalived and HAProxy

A single-node Kubernetes control plane is fine for a homelab until the node reboots for a firmware update and everything downstream of it stops. Once I had more than one Raspberry Pi to spare, the fix was the same pattern used for any fleet control plane: put a virtual IP in front of multiple API servers, health-check it, and fail over automatically.

The bootstrap command, health check, keepalived config, and HAProxy config, as a gist:

A few things worth walking through.

k3sup makes standing up k3s over SSH painless: k3sup install brings up the first node as the initial server, and k3sup join (with --server-ip pointed at that first node, which the note above was mid-sentence on) adds each additional one. --no-deploy traefik skips the bundled ingress controller; I'd rather choose that separately than inherit whatever k3s ships by default.

keepalived needs something to tell it whether a node's API server is actually healthy, not just pingable, which is what the check_apiserver.sh script is for. Both checks in it matter: the first confirms the local API server responds at all, and the second confirms the virtual IP is actually routing to a working node, not just assigned to a dead one.

In the keepalived config, fall 10 and rise 2 are deliberately asymmetric: it takes ten consecutive failed checks (30 seconds at a 3-second interval) to mark a node down, but only two successes to mark it back up. On a homelab Wi-Fi network, that asymmetry is what stops a single dropped packet from triggering a failover.

The HAProxy config runs in TCP mode, not HTTP: the API server terminates its own TLS, so HAProxy just needs to move bytes and health-check the port, not inspect the traffic.

What this actually buys you

None of this is exotic. It's the same VRRP-plus-load-balancer pattern used in front of any fleet-scale service, just running on four Raspberry Pis instead of production hardware. That's the point of building it at home: the failure modes (a node dropping off Wi-Fi, a health check flapping, an asymmetric fall/rise threshold tuned wrong) are the same ones that show up at any scale, and they're a lot cheaper to get wrong on a homelab than on a fleet.