Interview Prep Zoneby CuriouserLabs

Question 3 of 5 · staff level

What is split-brain, and how do quorums and fencing tokens prevent it?

🎤 Say this first

Split-brain is two nodes both believing they're the leader — typically after a partition or a false failure detection — each accepting writes, producing two divergent histories that cannot be merged. Majority quorums prevent it for the cluster's own state: since any two majorities intersect, only one side of a partition can gather a majority, and the minority side must stop. But that only protects the consensus system — the classic production failure is a leader that was paused (GC pause, VM freeze, network blip), lost its lease without noticing, and then wakes up and writes to external storage that has no idea it was demoted. The fix there is a fencing token: the lock service hands out a monotonically increasing number with every grant, and the downstream resource rejects any request carrying a token lower than the highest it has seen.

The full picture

Fencing: the storage layer, not the lock, is what enforces exclusivity
// Lock service returns a monotonically increasing token with the lease.
Lease lease = lockService.acquire("shard-42");   // token = 33

// ... a 20-second GC pause happens here. The lease expires.
// Another node acquires the lock and gets token = 34, and writes.

// We wake up believing we still hold the lock and issue a write:
storage.write("shard-42", data, lease.token()); // token 33

// Storage has already seen 34 -> rejects 33. Divergence prevented.
// WITHOUT the token check this write silently corrupts the shard,
// and the lock service did nothing wrong — it expired the lease correctly.
  • Say the quorum intersection property out loud: two majorities of the same set always share a member, so two conflicting decisions cannot both be ratified. Everything else about quorums is bookkeeping on top of that.
  • A lease is a lock with a clock, and clocks are the weak link: the holder believes it holds the lease until time T by its own clock, which may be skewed or frozen. This is why 'the lock expired so it's safe' is an unsound argument, and why fencing tokens exist — they move enforcement to the resource, which needs no clock at all.
  • STONITH / self-fencing is the blunter alternative: the demoted node is power-cycled or kills itself on lease loss. Effective in HA clusters, unavailable to you in most cloud/container setups — which is exactly why token-based fencing is the portable answer.
  • Witness/tiebreaker nodes solve the even-numbered-cluster problem cheaply: two full replicas plus a lightweight arbiter that holds no data but votes. Common in MongoDB (arbiter) and two-datacenter designs where a third DC gets only the witness.
  • The honest limitation: fencing only works if the downstream resource participates. S3, a plain filesystem, or a legacy service with no token support cannot reject a stale writer — in that case, no distributed lock is actually safe, and the design must change (single-writer partitioning, or conditional writes/ETags where available).

🔄 Likely follow-up questions

  • Why doesn't a lease TTL alone make a distributed lock safe?
  • Design fencing for a system whose storage is plain S3 — what can you actually use?
  • Why does a 4-node cluster tolerate no more failures than a 3-node one?