Sentinel — Automatic Failover Configuration

Replication gives us copies. Sentinel is what turns those copies into high availability — it watches the primary, and when enough Sentinels agree it is gone, it promotes a replica and reconfigures everyone else.

How Sentinel decides

Each Sentinel pings the primary. If it gets no reply within down-after-milliseconds, that Sentinel marks it subjectively down (SDOWN).
Sentinels gossip over port 26379. When quorum Sentinels agree, the primary is objectively down (ODOWN).
The Sentinels elect a leader among themselves (this needs a majority — 2 of 3), and the leader runs the failover: pick the best replica, send it REPLICAOF NO ONE to promote it, and point the others at it.

Two thresholds matter, and they are easy to conflate:

quorum — how many Sentinels must agree the master is down to start a failover.
majority — how many Sentinels must be reachable to authorise one.

With 3 Sentinels and quorum 2, you need 2 to detect failure and a majority (2) to act. Losing one node still leaves two — the cluster heals. This is exactly why the Sentinel count must be odd.

Configure Sentinel (all three nodes)

The redis-sentinel package ships /etc/redis/sentinel.conf with a default sentinel monitor mymaster 127.0.0.1 6379 2. Point it at the real primary and add auth + timers. Identical on all three nodes:

# all nodes
sudo sed -i 's|^sentinel monitor .*|sentinel monitor mymaster 10.100.100.101 6379 2|' /etc/redis/sentinel.conf

sudo tee -a /etc/redis/sentinel.conf >/dev/null <<'EOF'

# --- HA settings ---
bind 0.0.0.0 -::1
sentinel auth-pass mymaster ChangeMe_RedisPass
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 10000
sentinel parallel-syncs mymaster 1
EOF

sudo ufw allow from 10.100.100.0/24 to any port 26379 proto tcp
sudo systemctl enable --now redis-sentinel
sudo systemctl restart redis-sentinel

What each directive does:

Directive	Meaning
`sentinel monitor mymaster <ip> 6379 2`	Watch the primary named `mymaster`; quorum = 2
`sentinel auth-pass mymaster <pass>`	Password Sentinel uses to talk to Redis (required because we set `requirepass`)
`down-after-milliseconds … 5000`	No reply for 5 s → subjectively down
`failover-timeout … 10000`	Bounds retries / cooldown between failover attempts
`parallel-syncs … 1`	Re-sync replicas to the new primary one at a time (avoids a thundering-herd resync)

mymaster is just a label — one Sentinel fleet can monitor several primaries, each with its own name. Keep it consistent across all nodes.

Self-rewriting config: Sentinel edits its own config file at runtime to record what it discovers (replicas, peer Sentinels, the current primary). Expect extra lines to appear, and keep the file writable by the redis user (the package sets that up for you).

Verify the Sentinel quorum

redis-cli -p 26379 sentinel master mymaster

Look for:

name                   mymaster
ip                     10.100.100.101
flags                  master
num-slaves             2
num-other-sentinels    2
quorum                 2

num-slaves 2 and num-other-sentinels 2 mean this Sentinel sees both replicas and both peer Sentinels — the full 3-node quorum is formed and ready.