The full picture
- Frame it as the FLP consequence in operational clothes: unbounded message delay means no timeout is provably correct. So the honest engineering statement is not 'this node is dead' but 'I am willing to act as if it were dead, and here is the cost when I'm wrong.'
- Phi-accrual in one sentence: instead of a fixed threshold, model recent heartbeat intervals and output φ = −log₁₀(probability this delay is normal). φ=8 means 'roughly 1-in-10⁸ chance this is just slowness'. The value is that it self-tunes across a fast LAN and a congested WAN with no reconfiguration.
- Gossip/SWIM is how detection scales: all-to-all probing is O(n²) heartbeats. SWIM has each node probe a random peer, ask a few others to probe indirectly before declaring suspicion (which filters transient blips), and gossip the conclusion. Used by Consul, Serf, HashiCorp's stack, and Cassandra's gossip.
- Indirect probing is the underrated detail: 'I can't reach X — can you?' distinguishes X is down from my link to X is down, which is exactly the asymmetric-partition case that causes the nastiest incidents (each side thinks the other failed).
- Detection must be cheap and independent of the load path: a health endpoint that hits the database will report the node dead whenever the database is slow — turning one dependency's latency into a cluster-wide eviction storm. Liveness (is the process alive?) and readiness (should it get traffic?) are different checks, and Kubernetes separates them for exactly this reason.
| Timeout too short | Timeout too long |
|---|---|
| false failovers under GC/load spikes | slow recovery, longer outages |
| rebalancing storms (data moves needlessly) | clients keep hitting a dead node |
| split-brain risk if both sides act | stale leader keeps writing longer |
| cascades: slow → 'dead' → more load → slower | SLO breached on every real failure |