A gotcha that cost an afternoon

A war story, because the lessons that stick are the ones that bit you.

A backend's address was changed in the HAProxy GUI, Apply was clicked, and… HAProxy kept sending traffic to the old port. The config on disk said one thing; the running proxy did another. Blank pages, much confusion.

The culprit: HAProxy was configured to load server state from a file on start (load-server-state-from-file). On a graceful restart it restored the old server state — including address/port — from that stale file, overriding the freshly generated config. A normal restart couldn't fix it because the graceful stop rewrote the stale file first.

The fix was to stop HAProxy, delete the state file, then start it clean:

stop haproxy  ->  rm the server-state file  ->  start haproxy

Lesson learned: when a proxy's behaviour disagrees with its config, suspect cached/persisted state, not the config you're staring at. Server-state files, connection caches, and "saved" runtime state are exactly the kind of invisible layer that makes you doubt your own eyes. Knowing they exist is half the battle.