Preparing the nodes

Before kubeadm will touch a machine, the machine has to be ready. The same prep runs on all four nodes:

each node
  |- swap OFF            (kubelet refuses to run with swap by default)
  |- kernel modules:     overlay, br_netfilter
  |- sysctl:             net.ipv4.ip_forward = 1
  |                      net.bridge.bridge-nf-call-iptables = 1
  |- containerd          (the CRI runtime) with SystemdCgroup = true
  |- kubelet/kubeadm/kubectl  (held at the chosen version)

A few of these deserve a "why":

• Swap off. The kubelet wants a predictable relationship between a pod's memory request and real memory. Swap muddies that, so historically the kubelet just refuses to start with swap on. The VMs ship with a small swapfile, so part of node prep is turning it off and commenting it out of /etc/fstab.
• br_netfilter + ip_forward. Pods talk to each other across nodes through bridged/forwarded traffic. These settings let the kernel apply iptables rules to that traffic and actually forward it — without them, pod networking silently doesn't work.
• SystemdCgroup = true. The kubelet and the container runtime must agree on who manages cgroups. Ubuntu uses systemd as the cgroup manager, so containerd has to be told to match. Mismatch here causes kubelets that flap under load — a classic, maddening, hard-to-spot problem.
• Version pinning (apt-mark hold). Kubernetes components should move together and on purpose. Pinning stops an unattended apt upgrade from sliding kubelet out from under you.

Lesson learned: do this prep with one script run across every node, not by hand. I ran the four nodes in parallel from one place. Hand-prepping nodes is how you end up with one worker that's subtly different and a bug you chase for an hour.