Interview Prep Zoneby CuriouserLabs

Question 5 of 5 · architect level

When do you genuinely need atomic commit across services — and how do modern distributed databases get it without the blocking?

🎤 Say this first

Most 'distributed transactions' are a boundary mistake: two services writing data that one aggregate should own. The first move is always to check whether the invariant can live inside a single transaction — same database, same partition, single writer — because that's free and correct. When it genuinely can't (money moving between independently-owned ledgers, inventory across warehouses), you have three real options: a saga with compensations if the invariant tolerates a visible in-between; TCC / reservations if you can hold a claim while you decide; or a distributed database that does 2PC properly. And the trick those databases use is worth knowing: Spanner, CockroachDB, TiDB and YugabyteDB run 2PC where every participant is itself a Raft/Paxos group. A participant can't 'crash' — it fails over — and the coordinator's decision log is replicated, so the blocking window shrinks from until a human intervenes to one leader election.

The full picture

  • Ask the boundary question first and out loud: 'why are these two writes in different services?' If the answer is 'because we split by noun' or 'because a different team owns it', the fix is the service boundary, not a transaction protocol. An invariant that must hold atomically is the strongest possible signal that the data belongs to one aggregate — this is the single most valuable point you can make in a system-design interview on this topic.
  • Consensus doesn't make atomic commit non-blocking — replication does. These are two different problems that both use quorums: consensus makes one group agree on a value; atomic commit makes several groups agree to commit together, and it still needs 2PC on top. What changes is that a participant is now a fault-tolerant group rather than a single machine, so 'the participant is unreachable' stops being a permanent state. Being able to separate those two ideas is what distinguishes this answer.
  • The latency bill is unavoidable and worth quoting: a cross-shard transaction in one of these databases costs the 2PC round trips plus a Raft commit at every participant, so single-region cross-shard is roughly 2–4× a single-shard write, and cross-region is dominated by speed of light. That's exactly why they all give you tools — interleaved tables, locality-aware partitioning, REGIONAL BY ROW — to keep transactions inside one shard or one region. The optimization is always to avoid distributing the transaction, not to make it faster.
  • Where atomicity is genuinely non-negotiable: double-entry accounting (debit and credit must land together or the ledger doesn't balance), the last unit of limited inventory, regulatory records, and uniqueness on something with legal weight. For these, either colocate the data so it's one local transaction, or accept a real distributed transaction and its cost. A saga here means someone will eventually reconcile by hand.
  • Bounded-scope transactions are the pragmatic middle ground most teams should reach for: DynamoDB TransactWriteItems (all-or-nothing across up to 100 items in one region), MongoDB multi-document transactions, Postgres inside one instance, or a Kafka transaction for consume→process→produce within Kafka. Each is atomic within a boundary the platform can enforce cheaply. Design so your invariants fit inside one of those boundaries and the whole question disappears.
SituationReach forWhy
Two writes, one aggregateone local transactionthe boundary was wrong — fix that, not the protocol
State change + eventtransactional outboxatomic locally; at-least-once downstream
Multi-step business flowsaga + compensationsno blocking; you own the isolation
Hold a claim while decidingTCC / reservation with TTLexpiry replaces the compensation
Cross-shard invariant, one platform2PC over Raft groups (Spanner, CRDB, TiDB)blocking bounded by leader election
Cross-org / third-party APIsaga — nothing else existsyou cannot enlist someone else's system

🔄 Likely follow-up questions

  • Why doesn't running 2PC over Raft groups make atomic commit non-blocking in theory?
  • How would you model double-entry accounting across two services that must stay separate?
  • What does Spanner's INTERLEAVE IN PARENT buy you, and what does it cost?