Why we designed it this way

The cluster makes a few structural choices on purpose:

• Two vSwitches, not one. One private (cluster heartbeat plus VM east-west traffic), one public (inbound). Keeping the latency-sensitive cluster chatter on its own network means a busy public link can't disturb it.
• Corosync on the private network. Corosync is the cluster's heartbeat — how nodes agree on who's alive. It is sensitive to latency and jitter. Put it on a quiet, dedicated link and a traffic spike on the public side can't trigger false "node down" events (and the fencing dramas that follow).
• pfSense as the cluster-wide gateway. With one gateway serving VMs on any node, a VM can live on — or migrate to — any host and still reach the world through the same controlled edge, without rewiring its networking.
• Plan density around the per-node MAC ceiling. The provider caps how many MAC addresses a host may present. On a virtualisation host that ceiling — not CPU or RAM — can be the real limit on how many VMs you can run. Design with the number in mind.

And the bigger "why a cluster at all": three nodes give you live migration, a path to high availability, and one management plane across all of them — at the cost of more networking complexity, which is what this book is mostly about taming.

The mindset: separate the traffic that must stay healthy (cluster heartbeat) from the traffic that's noisy and bursty (public/VM). Most cluster instability traces back to those two sharing a path. Decide your real limits (here, MAC count) up front, instead of discovering them when you can't start VM number N.