Interview Prep Zoneby CuriouserLabs

Question 5 of 5 · architect level

You've accepted eventual consistency for a feature. How do you design it so users still perceive the system as correct?

🎤 Say this first

Perceived correctness comes from four moves. (1) Hide the lag with session guarantees — the writer reads their own write from the leader (or from local state) so they see truth immediately, while everyone else tolerates a short delay. (2) Make convergence a stated SLO, not a hope: measure replica lag, alarm on it, and design the UI around the p99. (3) Choose the conflict-resolution rule deliberately — last-write-wins silently discards data; prefer merges (CRDTs), append-only event logs, or explicit sibling resolution. (4) Make operations idempotent and commutative where you can, so replay and reorder are non-events. The failure mode to avoid is silent divergence: a system that converges to the wrong value with no way to detect it.

The full picture

  • Model the domain so conflicts can't exist, which beats resolving them: a like is a set insert (idempotent, commutative), a balance is a log of deltas rather than a mutable number, a counter is a G-Counter per replica summed on read. Most 'we need strong consistency' requirements dissolve once the data type stops being a mutable scalar.
  • Optimistic UI is a consistency decision in disguise: showing the write as done before it's durable is exactly read-your-writes implemented client-side. It's the right default if you also handle the rollback path — the reconciliation UX (a toast, a retry, a merge prompt) is the part teams forget, and its absence is what makes eventual consistency feel broken.
  • Bound the blast radius by scope: eventual across regions while linearizable within a region is a very common and defensible architecture — cross-region users tolerate seconds of lag; same-region users never see it. Choosing the boundary where the guarantee weakens is the real design work.
  • Detect divergence actively: anti-entropy (Merkle-tree repair like Cassandra/Dynamo), read repair, and periodic reconciliation jobs comparing source-of-truth to derived stores. If you can't answer 'how would you know two replicas disagreed?', you have eventual consistency without eventual convergence.
  • Some operations simply cannot be eventual: uniqueness constraints (usernames), limited inventory, and anything with regulatory or money semantics need coordination — either consensus, or a single-writer partition that owns the invariant. Say which of your operations are in this set and route them differently.
Data shapeConflict ruleRisk
Set membership (likes, tags)OR-Set CRDT / unionlow — merges are natural
Counter (views, votes)per-replica G-Counter, sum on readlow, but no exact real-time total
Text documentCRDT / OT (Yjs, Automerge)medium — library complexity
Mutable record (profile)LWW per field, HLC timestampsilent data loss on concurrent edits
Money / inventorydon't — use consensus or single-writercorrectness, not UX

🔄 Likely follow-up questions

  • How would you enforce unique usernames in an AP system?
  • What does anti-entropy with Merkle trees actually compare, and how often should it run?
  • Where does optimistic UI become dishonest, and what should the rollback look like?