The full picture
// Lock service returns a monotonically increasing token with the lease.
Lease lease = lockService.acquire("shard-42"); // token = 33
// ... a 20-second GC pause happens here. The lease expires.
// Another node acquires the lock and gets token = 34, and writes.
// We wake up believing we still hold the lock and issue a write:
storage.write("shard-42", data, lease.token()); // token 33
// Storage has already seen 34 -> rejects 33. Divergence prevented.
// WITHOUT the token check this write silently corrupts the shard,
// and the lock service did nothing wrong — it expired the lease correctly.- Say the quorum intersection property out loud: two majorities of the same set always share a member, so two conflicting decisions cannot both be ratified. Everything else about quorums is bookkeeping on top of that.
- A lease is a lock with a clock, and clocks are the weak link: the holder believes it holds the lease until time T by its own clock, which may be skewed or frozen. This is why 'the lock expired so it's safe' is an unsound argument, and why fencing tokens exist — they move enforcement to the resource, which needs no clock at all.
- STONITH / self-fencing is the blunter alternative: the demoted node is power-cycled or kills itself on lease loss. Effective in HA clusters, unavailable to you in most cloud/container setups — which is exactly why token-based fencing is the portable answer.
- Witness/tiebreaker nodes solve the even-numbered-cluster problem cheaply: two full replicas plus a lightweight arbiter that holds no data but votes. Common in MongoDB (arbiter) and two-datacenter designs where a third DC gets only the witness.
- The honest limitation: fencing only works if the downstream resource participates. S3, a plain filesystem, or a legacy service with no token support cannot reject a stale writer — in that case, no distributed lock is actually safe, and the design must change (single-writer partitioning, or conditional writes/ETags where available).