The full picture
- Frame it as the FLP consequence in operational clothes: unbounded message delay means no timeout is provably correct. So the honest engineering statement is not 'this node is dead' but 'I am willing to act as if it were dead, and here is the cost when I'm wrong.'
- Phi-accrual in one sentence: instead of a fixed threshold, model recent heartbeat intervals and output φ = −log₁₀(probability this delay is normal). φ=8 means 'roughly 1-in-10⁸ chance this is just slowness'. The value is that it self-tunes across a fast LAN and a congested WAN with no reconfiguration.
- Gossip is how detection scales: all-to-all probing is O(n²) heartbeats. SWIM (HashiCorp's memberlist, so Serf, Consul and Nomad) has each node probe a random peer, ask a few others to probe indirectly before declaring suspicion, and gossip the conclusion. Cassandra gossips too but is not SWIM — it's Scuttlebutt-style anti-entropy exchange with a phi-accrual detector and no indirect probing. Both beat O(n²); only SWIM filters transient blips via indirection.
- Indirect probing is the underrated detail: 'I can't reach X — can you?' distinguishes X is down from my link to X is down, which is exactly the asymmetric-partition case that causes the nastiest incidents (each side thinks the other failed).
- Detection must be cheap and independent of the load path: a health endpoint that hits the database will report the node dead whenever the database is slow — turning one dependency's latency into a cluster-wide eviction storm. Liveness (is the process alive?) and readiness (should it get traffic?) are different checks, and Kubernetes separates them for exactly this reason.
| Timeout too short | Timeout too long |
|---|---|
| false failovers under GC/load spikes | slow recovery, longer outages |
| rebalancing storms (data moves needlessly) | clients keep hitting a dead node |
| split-brain risk if both sides act | stale leader keeps writing longer |
| cascades: slow → 'dead' → more load → slower | SLO breached on every real failure |